DEV Community: Matt Lenhard

Comprehensive Guide to Monitoring in MCP

Matt Lenhard — Sun, 04 May 2025 18:41:01 +0000

The Ultimate Guide to Monitoring in Model Context Protocol

In the rapidly evolving landscape of AI agent systems, the Model Context Protocol (MCP) has emerged as a powerful standard for enabling AI models to interact with tools and external resources.

As organizations increasingly deploy MCP servers in production environments, robust monitoring becomes not just beneficial but essential.

This guide explores how to implement comprehensive monitoring for your MCP deployments to ensure reliability, performance, and security.

I've recently been building a solution for mcp monitoring and wanted to type up some of what I've learned.

Understanding MCP-Specific Monitoring Challenges

Monitoring MCP deployments presents unique challenges that differ from traditional application monitoring. Unlike simpler systems, MCP involves complex tool execution flows where servers expose tools that AI models can invoke, requiring monitoring of both the request and execution outcome.

The protocol relies on stateful, asynchronous communication patterns with long-lived connections between clients and servers. Additionally, the MCP protocol itself generates specific messages and events that require dedicated monitoring approaches.

MCP's support for diverse transport mechanisms (stdio, HTTP/SSE) adds another layer of complexity, as each requires different monitoring strategies.

Perhaps most critically, since MCP grants AI systems access to tools and resources, security monitoring becomes paramount to prevent misuse and protect sensitive operations.

A complete MCP monitoring strategy must account for these unique characteristics while still covering the fundamentals of system health and performance.

Key MCP-Specific Metrics to Monitor

Protocol Message Metrics

MCP operates through a defined message protocol. Key metrics to track include message volume by type, which helps identify unusual patterns in how AI models interact with your MCP servers.

By monitoring the frequency of different message types—tool discovery requests, tool execution requests, resource access requests, initialization messages, and error responses—you can establish baseline patterns and quickly spot anomalies.

Protocol version distribution tracking provides visibility into which MCP protocol versions clients are using, helping identify outdated clients or compatibility issues that might affect stability.

Message size distribution monitoring is equally important, as oversized messages can indicate inefficient usage patterns or potential abuse, especially for resource requests that might return large datasets.

Tool Execution Metrics

The heart of MCP is tool execution, which requires specialized metrics focused on performance and reliability. Tool popularity tracking shows which tools are most frequently invoked, helping prioritize optimization efforts where they'll have the most impact.

Tool execution time monitoring should break down performance into key phases: tool initialization time, actual execution duration, and response formatting time.

This granular view helps pinpoint bottlenecks in your implementation.

Error rates should be tracked by tool type and error category, such as permission denied errors, invalid parameter errors, timeouts, and internal execution failures.

By categorizing these errors, you can identify patterns that might indicate configuration issues, client misunderstandings, or security concerns.

Parameter pattern analysis extends this further by revealing common usage patterns, potential misuse scenarios, or opportunities for optimization based on how AI models typically interact with your tools.

Resource Access Patterns

For MCP servers that expose resources like files or data, monitoring access patterns becomes crucial for both performance and security.
Resource access frequency metrics reveal which resources are in highest demand, allowing you to optimize caching strategies or distribution methods for frequently accessed content.

Resource response size tracking helps identify potential performance issues related to large data transfers that might affect client-side processing or network bandwidth.

Perhaps most importantly, detecting access pattern anomalies can reveal security issues or inefficient AI behavior that might otherwise go unnoticed.

Sudden changes in which resources are accessed, unusual access sequences, or access attempts outside normal patterns can all indicate potential security issues that require immediate investigation.

Implementing MCP-Specific Monitoring

Instrumenting Protocol Message Flow

To effectively monitor MCP protocol messages, implement tracing at the transport layer:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import { metrics } from "./monitoring.js"; // Your metrics library (Prometheus, Datadog, etc.)

// Create an MCP server with monitoring
const server = new McpServer({
  name: "MonitoredMcpServer",
  version: "1.0.0"
});

// Create a custom transport that wraps the standard transport
class MonitoredTransport extends StdioServerTransport {
  constructor() {
    super();

    // Ensure original methods are preserved
    const originalReceive = this.receive.bind(this);
    const originalSend = this.send.bind(this);

    // Override receive to monitor incoming messages
    this.receive = async () => {
      const message = await originalReceive();
      if (message) {
        const messageType = message.method || 'response';
        metrics.increment('mcp.message_received', { type: messageType });
        metrics.histogram('mcp.message_size', JSON.stringify(message).length);
      }
      return message;
    };

    // Override send to monitor outgoing messages
    this.send = async (message) => {
      const messageType = message.method || 'response';
      metrics.increment('mcp.message_sent', { type: messageType });
      metrics.histogram('mcp.message_sent_size', JSON.stringify(message).length);
      return originalSend(message);
    };
  }
}

// Start receiving messages with monitoring
const transport = new MonitoredTransport();
await server.connect(transport);

Tracking Tool Execution

Instrument your tool execution handlers to capture detailed performance and usage metrics:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import { metrics } from "./monitoring.js";

// Create an MCP server
const server = new McpServer({
  name: "DatabaseToolServer",
  version: "1.0.0"
});

// Add a monitored database query tool
server.tool("executeQuery",
  { 
    query: z.string(), 
    parameters: z.record(z.any()).optional() 
  },
  async ({ query, parameters = {} }) => {
    // Track tool invocation
    metrics.increment('mcp.tool.invoked', { tool: 'executeQuery' });

    // Start timing the execution
    const timer = metrics.startTimer('mcp.tool.execution_time');

    try {
      // Validate parameters
      if (!isValidQuery(query)) {
        metrics.increment('mcp.tool.invalid_parameters', { tool: 'executeQuery' });
        return {
          content: [{ 
            type: "text", 
            text: "Error: Invalid query format"
          }]
        };
      }

      // Execute the tool logic
      const result = await database.runQuery(query, parameters);

      // Track result size
      const resultString = JSON.stringify(result);
      metrics.histogram('mcp.tool.result_size', resultString.length);

      metrics.increment('mcp.tool.success', { tool: 'executeQuery' });

      return {
        content: [{ 
          type: "text", 
          text: resultString 
        }]
      };
    } catch (error) {
      metrics.increment('mcp.tool.error', { 
        tool: 'executeQuery',
        error_type: error.name
      });

      return {
        content: [{ 
          type: "text", 
          text: `Error: ${error.message}` 
        }]
      };
    } finally {
      timer.end({ tool: 'executeQuery' });
    }
  }
);

// Start receiving messages
const transport = new StdioServerTransport();
await server.connect(transport);

Monitoring Resource Access

For MCP servers that expose resources, implement specific monitoring:

import { McpServer, ResourceTemplate } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { metrics } from "./monitoring.js";

// Create an MCP server
const server = new McpServer({
  name: "FileServer",
  version: "1.0.0"
});

// Add a monitored file resource
server.resource(
  "data",
  new ResourceTemplate("file://{filename}", { list: undefined }),
  async (uri, { filename }) => {
    const resourceId = `file://${filename}`;
    metrics.increment('mcp.resource.accessed', { resource: resourceId });

    const timer = metrics.startTimer('mcp.resource.access_time');

    try {
      const data = await fs.readFile(filename, 'utf-8');

      // Track resource size
      metrics.histogram('mcp.resource.size', data.length);
      metrics.increment('mcp.resource.success', { resource: resourceId });

      return {
        contents: [{
          uri: uri.href,
          text: data
        }]
      };
    } catch (error) {
      metrics.increment('mcp.resource.error', { 
        resource: resourceId,
        error_type: error.name
      });

      throw error;
    } finally {
      timer.end({ resource: resourceId });
    }
  }
);

// Start receiving messages
const transport = new StdioServerTransport();
await server.connect(transport);

MCP-Specific Alerting Strategies

Effective alerting for MCP servers must focus on issues unique to the protocol.

Security monitoring should be a primary concern, with alerts configured for unauthorized tool access attempts, anomalous tool usage patterns, resource access violations, and malformed protocol messages that don't conform to the MCP specification.

These security-focused alerts help protect your systems from potential attacks or misuse by detecting suspicious activities early.

Performance monitoring is equally important for maintaining a responsive system.

Configure alerts for tool execution time thresholds when tools exceed expected execution times, resource response size anomalies that could impact performance, and connection lifetime anomalies when connections stay open too long or close prematurely.

These alerts help maintain optimal performance by flagging issues before they significantly impact users.

Reliability alerts complete the picture by monitoring for tool failure rate thresholds, protocol error spikes that might indicate client or server issues, and connection establishment failures that could signal problems with initial MCP handshakes.

By implementing this comprehensive alerting strategy, you can ensure your MCP deployment remains secure, performant, and reliable even as usage patterns evolve and scale.

Advanced MCP Monitoring Techniques

Session Tracing

Implement session-level tracing to track the entire lifecycle of MCP connections:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import { logger, metrics } from "./monitoring.js";

// Create an MCP server with session handling
const server = new McpServer({
  name: "SessionTrackedServer",
  version: "1.0.0"
});

// Custom transport wrapper for session tracking
class SessionTrackingTransport extends StdioServerTransport {
  constructor() {
    super();
    this.sessionId = generateSessionId();

    logger.info('MCP session started', { sessionId: this.sessionId });
    this.timer = metrics.startTimer('mcp.session.duration');

    // Track session in connection events
    this.on('connect', () => {
      metrics.increment('mcp.session.connected', { 
        sessionId: this.sessionId,
        transport: 'stdio'
      });
    });

    this.on('disconnect', () => {
      metrics.increment('mcp.session.disconnected', { 
        sessionId: this.sessionId,
        transport: 'stdio'
      });
      this.timer.end();
      logger.info('MCP session ended', { sessionId: this.sessionId });
    });

    // Add session context to all monitoring events
    const originalReceive = this.receive.bind(this);
    this.receive = async () => {
      const message = await originalReceive();
      if (message) {
        metrics.increment('mcp.message_received', { 
          sessionId: this.sessionId,
          type: message.method || 'response'
        });
      }
      return message;
    };
  }
}

// Start session tracking
const transport = new SessionTrackingTransport();
await server.connect(transport);

Tool Usage Analytics

Beyond basic metrics, advanced analytics on tool usage patterns can reveal deeper insights into how AI models leverage your MCP implementation.

Tool chain analysis identifies common sequences of tool calls, allowing you to optimize frequently used workflows for better performance and reliability.

Parameter distribution monitoring tracks the statistical patterns in how tools are used, helping detect unexpected shifts in input patterns that might indicate model behavior changes or potential issues.

Tool dependency mapping takes this analysis further by documenting relationships between different tools and resources, helping you understand how failures might cascade through your system.

This holistic view of tool interdependencies is invaluable for improving system resilience and planning maintenance windows that minimize disruption to critical workflows.

Security Anomaly Detection

Machine learning-based anomaly detection specific to MCP interactions represents the cutting edge of monitoring for these systems.

By establishing baselines of normal behavior for tool usage, resource access, and message flow, you can train models to detect subtle deviations that might indicate security issues or performance problems.

These anomaly detection systems become increasingly valuable as your MCP deployment grows in complexity, catching issues that would be difficult to identify with traditional threshold-based monitoring.

The most advanced implementations correlate anomalies across different aspects of the system—connecting unusual protocol messages with atypical tool execution patterns and unexpected resource access—to build a comprehensive picture of potential security incidents.

This correlation greatly reduces false positives and provides security teams with the context they need to quickly understand and address emerging threats.

MCP-Specific Monitoring Dashboards

Creating dedicated dashboards for MCP monitoring highlights the protocol's unique aspects and provides at-a-glance visibility into system health.

A protocol health dashboard should display message flow rates by type, protocol error counts and types, client version distribution, and message size distributions—all key indicators of the communication layer's health.

The tool execution dashboard should focus on operational performance metrics like tool invocation rates, execution times (p50, p95, p99), error rates by tool, and popular tool chains.

These metrics help operations teams quickly identify performance bottlenecks or reliability issues in specific tools.

For resource management, a resource access dashboard should present access patterns, response sizes, error rates, and access latency. This visibility is particularly important for systems that manage sensitive or high-value resources, as it helps detect both performance issues and potential security concerns related to how resources are being accessed.

Conclusion

Effective monitoring of MCP deployments requires understanding the protocol's unique characteristics and challenges.

By focusing on protocol messages, tool execution, and resource access patterns, you can build a comprehensive monitoring solution that ensures reliability, performance, and security.

The most successful MCP implementations treat monitoring as an evolving practice.

As your MCP ecosystem grows with more tools and resources, your monitoring approach should adapt to maintain visibility into increasingly complex interactions between AI models and your systems.

By implementing the MCP-specific monitoring patterns outlined in this guide, you'll be well-positioned to operate reliable and secure AI agent systems in production environments.

MCP Evals - How to Test Your MCP Tools

Matt Lenhard — Sun, 27 Apr 2025 13:43:39 +0000

MCP Evals: A Deep Dive into Evals in MCP

As LLMs power more products, systematic evaluation becomes essential to ensure reliability and user satisfaction. Evals—short for evaluations—are structured processes that measure how well a model meets predefined criteria. This article explores what evals are, why they matter, and how to design and run them effectively.

What Are Evals?

Evals refer to the assessments, both automated and human-driven, that quantify an AI model's performance on specific tasks. Unlike traditional unit tests, evals capture qualitative aspects such as:

Accuracy: Does the model produce correct information?
Completeness: Does it cover all necessary details?
Relevance: Is the output appropriate for the user's query?
Clarity: Is the response easy to understand?
Reasoning: Does the model demonstrate logical steps?

Evals help answer questions like:

How often does the model hallucinate facts?

Is the tone suitable for the target audience?

Can the model consistently use external tools via protocols like MCP?

Why Evals Matter for MCP

In MCP (Model Context Protocol) workflows, AI assistants rely on external tools exposed by servers to perform tasks (MCP Introduction). Evals ensure these tool integrations work correctly and consistently by:

Verifying correct tool selection: Evals check that the model invokes the appropriate tool for each query.
Validating input schemas: They confirm tool arguments conform to the required JSON schema, preventing runtime errors (MCP Tools).
Testing error handling: Evals assess how the model and server respond to invalid inputs or tool failures.
Measuring end-to-end behavior: They evaluate the entire MCP round-trip, from tool discovery to invocation results.

Types of Evals

Evals generally fall into three main categories:

Human Evaluations

Subject-matter experts or end users rate outputs using thumbs-up/down, numerical scales, or feedback forms. This yields high-quality judgments but can be costly and sparse.
Code-Based Evaluations

Automated scripts check outputs against objective criteria—for example, verifying that generated code compiles without errors or that JSON outputs match a schema. These evals are fast and repeatable but may miss subtleties.
LLM-Based Evaluations

A secondary "judge" model scores or labels outputs based on prompts. This approach offers scalable, detailed feedback at lower cost, though it requires careful prompt design and calibration.

Designing an Effective Eval

Each eval typically follows four steps:

Set the Role: Define the judge's perspective (e.g., "You are a technical reviewer.").
Provide Context: Supply the model output and any supporting information.
Specify the Goal: Clearly articulate what success looks like (e.g., "Rate factual accuracy on a scale of 0–1.").
Define Labels: Explain the scoring rubric or categories.

Example: Automated Accuracy Eval

npm install mcp-evals

Creating Evaluations

Here's a simple example of how to create an evaluation via the mcp evals package (I'm the author).

import { EvalConfig } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade, EvalFunction} from "mcp-evals";

const weatherEval: EvalFunction = {
    name: 'Weather Tool Evaluation',
    description: 'Evaluates the accuracy and completeness of weather information retrieval',
    run: async () => {
      const result = await grade(openai("gpt-4"), "What is the weather in New York?");
      return JSON.parse(result);
    }
};

const config: EvalConfig = {
    model: openai("gpt-4"),
    evals: [weatherEval]
};

export default config;

Running Evals in CI

  name: AI Evals
  on: [push]
  jobs:
    evaluate:
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v3
        - name: Install Dependencies
          run: npm install mcp-evals openai
        - name: Run Evals
          run: npx mcp-eval src/evals.ts src/server.ts

For full documentation on how to integrate evals into your mcp server see the documentation here.

Best Practices

Start simple: Begin with a few core evals before scaling up.
Mix eval types: Combine human, code-based, and LLM-based methods.
Iterate on prompts: Refine your judge-prompts to reduce ambiguity.
Track trends: Use dashboards to monitor eval scores over time.

Conclusion

Evals are the backbone of reliable AI applications. By defining clear criteria and automating assessments, you can detect regressions early, maintain quality, and build user trust. Whether you write human surveys, code checks, or LLM-judge prompts, systematic evals ensure your AI delivers on its promises.