MCP Servers Keep Breaking in Production. Here's the 4 Failure Modes I Track Now.

Every few weeks, one of my MCP servers starts returning garbage. Not errors — worse. Silent corrupted data. Responses that look correct until you look at the timestamps, or the IDs, or the file paths.

I've been running MCP servers in production for about a year now. After the third or fourth incident, I stopped treating each failure as a one-off and started tracking patterns. This is what I've actually seen fail, and what I do about it now.

Failure Mode 1: The Stale Context Trap

MCP servers maintain state between calls. That's the point — they remember what file you were editing, what commit you were on, what Slack channel you queried. But that state lives in memory, and it accumulates silently.

The symptom: your server starts returning data that's technically valid but contextually wrong. A file read returns content from three requests ago. A database query returns rows that matched the first query, not the current one.

Here's the thing nobody tells you: the LLM doesn't know this is happening. It will confidently reason from corrupted context and produce output that looks perfectly logical.

The fix I've settled on:

# Wrap every MCP tool call with explicit context management
async def call_mcp_tool(server, tool_name, params, context_id=None):
    """Each call gets an explicit context marker."""
    import uuid
    ctx = context_id or str(uuid.uuid4())[:8]

    response = await server.call(tool_name, {**params, "_context": ctx})

    # Validate: did we get back what we asked for?
    if not validate_response(response, params):
        raise ContextStaleError(f"Server returned stale context for {ctx}")

    return response

The _context marker is a hack — but it lets the server verify which request a response actually belongs to.

Failure Mode 2: Timeout Cascades

MCP calls look cheap because they're fast in demos. In production, the same calls can take 10x longer when the remote service is under load.

The problem: most MCP client implementations set a fixed timeout, say 30 seconds. When that timeout fires, the LLM doesn't get an error it can reason about — it gets a generic timeout, retries the same call, and now you have two in-flight requests for the same operation.

This is how you get double-bookings, duplicate file writes, or the same Slack message sent three times.

My current mitigation is a circuit breaker pattern with exponential backoff:

from asyncio import sleep
from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=2, min=1, max=30)
)
async def robust_call(tool_fn, *args, **kwargs):
    try:
        return await tool_fn(*args, **kwargs)
    except TimeoutError as e:
        # Log but let tenacity handle the retry
        logger.warning(f"Timeout on {tool_fn.__name__}, retrying...")
        raise
    except RateLimitError:
        # Longer backoff for rate limits
        await sleep(60)
        raise

Notice RateLimitError gets its own branch. Most MCP implementations don't differentiate between timeout types, but the remote services behind them almost always have rate limits — and hammering a rate-limited service makes it worse.

Failure Mode 3: The Schema Drift Problem

MCP servers evolve. A tool that previously returned {id, name, status} starts returning {id, name, status, metadata, tags}. Or a required parameter gets added. Or a field type changes from string to string[].

The LLM using the tool doesn't know this happened. It keeps sending the old parameter shape. The server either silently ignores unknown fields, or throws a validation error that the LLM interprets as a transient failure and retries.

The drift accumulates over days. You don't notice until a quarterly audit.

I've started running schema validation as a nightly check:

import jsonschema

EXPECTED_SCHEMAS = {
    "filesystem.read_file": {
        "type": "object",
        "required": ["path"],
        "properties": {"path": {"type": "string"}}
    },
    "github.get_commit": {
        "type": "object",
        "required": ["owner", "repo", "sha"],
        "properties": {
            "owner": {"type": "string"},
            "repo": {"type": "string"},
            "sha": {"type": "string"}
        }
    }
}

def validate_server_schemas(server):
    """Run daily to catch schema drift before it bites."""
    for tool_name, schema in EXPECTED_SCHEMAS.items():
        tool = server.get_tool(tool_name)
        if not tool:
            logger.error(f"Tool {tool_name} missing from server")
            continue

        try:
            jsonschema.validate(tool.input_schema, schema)
        except jsonschema.ValidationError as e:
            logger.warning(f"Schema drift detected: {tool_name} — {e.message}")

This won't catch every drift scenario, but it catches the breaking changes before they hit a running agent.

Failure Mode 4: Tool Call Loops

The most expensive failure mode. The LLM gets a response that it interprets as incomplete, calls the same tool again with a slightly different parameter, gets another incomplete response, and loops.

I've seen loops run 40+ times before hitting a hard limit. Each iteration costs API tokens, API calls, and — if you're logging — significant storage. One looping incident cost me $23 in OpenAI calls before I caught it.

The fix isn't sophisticated:

MAX_CALLS_PER_TASK = 50

async def tracked_call(server, tool_name, params, call_count=0):
    if call_count >= MAX_CALLS_PER_TASK:
        raise LoopDetectedError(
            f"Exceeded {MAX_CALLS_PER_TASK} calls for {tool_name}. "
            "Possible loop detected."
        )

    result = await server.call(tool_name, params)

    # Simple heuristic: if the last 3 calls had identical structure but
    # different inputs, flag as potential loop
    if detect_loop_pattern(result, call_count):
        raise LoopDetectedError(f"Loop pattern detected at call {call_count}")

    return result

detect_loop_pattern looks at the last N responses and checks if they're structurally identical with incrementally different IDs or timestamps — a reliable loop signal.

What I Learned

MCP servers are genuinely useful. The reliability problems aren't fundamental flaws — they're the same problems any distributed system faces, just wearing LLM-friendly clothing.

The patterns that actually help:

1. Explicit context over implicit state. If the server won't give you context markers, inject them yourself.
2. Typed timeouts with different handlers. Don't lump timeouts and rate limits together.
3. Schema validation as a scheduled task. Not just at startup.
4. Hard call limits with loop detection. The LLM will retry. You need to be the one who says stop.

The fifth thing I've learned: write the monitoring before you need it. The failures I'm describing all look fine in the first week. It's week three that things start quietly degrading.

I'd rather ship a dashboard than debug a production incident at 2 AM.