I Audited 12 MCP Servers From My Agent Stack. 4 of Them Were Quietly Breaking.

I was running 12 MCP servers across two production agents when I noticed one of my "working" tools had been returning malformed JSON for six days.

No error. No log. No alert. The agent kept calling it, kept getting garbage back, kept asking me "did that work?" like everything was fine.

That was the moment I started actually auditing MCP servers instead of just running them. Here's what I found.

The Audit Setup

I treated this like a smoke test, not a benchmark. Three things I wanted to know per server:

1. Schema honesty — does the server's declared inputSchema match what it actually accepts?
2. Error surfacing — when the server fails, does the agent find out, or does it silently get a half-truth?
3. Retry safety — is the operation idempotent enough that the agent can retry without making things worse?

I wrote a small harness that hits each server with a battery of probe calls: a known-good input, a known-bad input (wrong types, missing fields, oversized strings), and a "natural" input that mimics what an LLM would actually pass when it's being lazy about schema details.

import asyncio
from dataclasses import dataclass

@dataclass
class ProbeResult:
    server: str
    tool: str
    case: str
    status: str          # "ok", "schema_mismatch", "silent_failure", "ok_but_lie"
    notes: str = ""

async def audit_tool(session, server_name, tool_name, schema):
    results = []

    # 1. Known-good input
    ok_input = build_typical_input(schema)
    r = await session.call_tool(server_name, tool_name, ok_input)
    results.append(evaluate(r, expected_shape=ok_input))

    # 2. Missing optional field
    partial = drop_optionals(ok_input)
    r = await session.call_tool(server_name, tool_name, partial)
    results.append(evaluate(r, expected_shape=ok_input))

    # 3. Wrong type on a field
    bad = ok_input.copy()
    for k in bad:
        bad[k] = str(bad[k])  # coerce everything to string
    r = await session.call_tool(server_name, tool_name, bad)
    results.append(evaluate(r, expected_shape=ok_input))

    return results

I expected a few rough edges. I did not expect what I got.

The Four Failures

1. The silent type coercer (schema drift)

One file-search server documented path as string. The agent passed /tmp/foo.txt and got back results. Great.

Then it passed a list of paths (because the LLM decided batching would be efficient), and the server quietly joined them into a single string and ran a glob against /tmp/foo.txt,/tmp/bar.txt. It returned matches for paths that didn't exist on disk. The agent then reported those matches to me as "files I found."

The schema said string. The server accepted array. The deviation wasn't an error — it was a feature, undocumented. Fix: add a JSON Schema validator in front of every tool call.

2. The "success" response that wasn't

A scheduling server returned {"status": "ok"} on every code path — including the one that meant "API key invalid" and the one that meant "user quota exceeded." It was using status: ok as a wrapper, not a real result indicator.

My agent reads status: ok and moves on. Every time. The real error was three layers deeper in the response body, and the agent never looked.

The fix was annoying: I had to add a per-tool error-projection layer that knew the real success criteria for each tool. There's no general fix for this — the agent has to know what's a lie.

3. The non-idempotent retry trap

A "send notification" tool documented itself as safe to retry. The server's response said message_id. Calling it twice with the same payload would, in theory, just give me back the same message_id.

In practice: it sent the message twice. Every time. The message_id returned was the latest one, not a dedupe key. My retry loop, which I'd tuned to be aggressive because "this tool says it's safe to retry," was spamming users.

Lesson: "idempotent" in MCP server docs means the author hopes it's idempotent, not that it is. Probe the actual behavior.

4. The MCP server that only worked at noon

This one was a clock-skew bug. A tool that fetched timestamps compared its own clock to a remote service clock, and the comparison was off by 13 seconds on a 12-hour cycle. For most of the day it worked. For a 4-minute window each afternoon, every call returned an expired_token error.

The agent retried. The retry hit the same 4-minute window and failed again. The agent gave up and reported "auth broken, please investigate" — which I did, and which took me an embarrassing afternoon to track down.

The Probe Patterns That Caught These

Three patterns did most of the work. None of them are exotic.

# 1. Type-coercion probe
def test_type_coercion(schema, server_call):
    sample = build_typical_input(schema)
    for field, spec in schema["properties"].items():
        if spec.get("type") == "integer":
            mutated = sample | {field: "not_a_number"}
            r = server_call(mutated)
            if not r.is_error:
                return f"FAIL: server accepted non-int for {field}"

# 2. Status field probe
def test_status_field(server_call, real_success_payload):
    # Pass input that should clearly fail; check if status still says ok
    r = server_call(make_garbage_input())
    if r.body.get("status") == "ok" and not real_success_payload(r):
        return "FAIL: success status reported on error path"

# 3. Idempotency probe (run twice, compare side effects)
def test_idempotency(server_call, side_effect_counter):
    side_effect_counter.reset()
    server_call(input_a)
    n1 = side_effect_counter.value
    server_call(input_a)
    n2 = side_effect_counter.value
    if n2 - n1 > 1:
        return f"FAIL: {n2 - n1} side effects from 2 identical calls"

What I Changed

After the audit, four things went into my agent runtime:

1. Per-tool error projection. Every tool call goes through a wrapper that knows the real success shape for that specific tool. Generic status: ok is no longer trusted.
2. Schema enforcement at the agent boundary. Tools that don't reject obviously wrong inputs get a pre-processor that does. Sometimes the pre-processor is the only thing keeping the agent honest.
3. Idempotency by default for retry logic. I only retry when I have a real idempotency key (or no side effects to worry about). "The server docs say it's safe" stopped counting.
4. Clock-skew probes for any time-sensitive tool. A 30-second run at different hours catches most of these.

What I Learned

The MCP ecosystem is moving fast and most servers are written by people who are solving their own problem first and documenting it second. That's fine for prototypes. It's a problem when the agent is the only user, because the agent can't ask "wait, that response doesn't make sense" the way a human would.

If you're running MCP servers in production, assume three things:

• The schema is a suggestion.
• The error reporting is whatever the author thought of at 2am.
• The idempotency story is optimistic.

Audit them. Your agent will thank you, even if it can't say so.

---

If you want the full audit harness I used (about 300 lines of Python), I've started packaging it up. Hit me up — happy to share once it's not still half-broken.