I Wrote Integration Tests for My MCP Failure Library. Here's the Pattern That Caught 3 Hidden Bugs.

I'd been shipping fixes to my MCP failure library for weeks. Every release felt solid. Then I've written one real integration test - and three "fixed" bugs weren't fixed at all.

The Problem

My failure library is an MCP server that stores crash patterns so AI agents can warn each other before hitting the same bug twice. Think of it as shared memory for tool errors: agent A hits a timeout talking to an external API, records the failure pattern, and agent B checks the library before making the same call. It works - when the server is running and the database is primed.

But getting it to work reliably in production was a different story. I'd been testing each component in isolation: the SQLite layer with unit tests, the MCP transport with mock clients, the retry logic with controlled timeouts. Each passed with flying colors. I had 90%+ coverage.

And yet, in production, things broke every other day. A fix that worked locally would fail in CI. A feature that passed every unit test would crash the first time two agents used it simultaneously.

The issue wasn't the code - it was the Interactions. A tool call that succeeded in isolation could fail when the session tracking layer hadn't initialized. A fix that worked with one MCP client crashed with another. Unit tests proved individual functions were mathematically correct, but they couldn't prove the system worked as a whole.

The Fix: Integration Test Harness

I wrote a short integration test harness that spins up the real MCP server, connects a real client, and runs end-to-end scenarios. No mocks, no stubs, no fakes - the actual STDIO transport, the real SQLite database, the production retry logic.

import subprocess
import json
import time
import sys

def call_tool(server, name, args=None):
    request = {
        "jsonrpc": "2.0",
        "id": int(time.time() * 1000),
        "method": "tools/call",
        "params": {"name": name, "arguments": args or {}}
    }
    server.stdin.write(json.dumps(request) + "\n")
    server.stdin.flush()
    line = server.stdout.readline()
    return json.loads(line)

server = subprocess.Popen(
    ["python", "-m", "mcp_failure_server", "--stdio"],
    stdin=subprocess.PIPE, stdout=subprocess.PIPE,
    stderr=subprocess.PIPE, text=True
)

record = call_tool(server, "record_failure", {
    "tool": "browser_navigate", "error": "Connection refused"
})
assert record.get("id")

query = call_tool(server, "query_similar", {
    "tool": "browser_navigate", "error_fragment": "refused"
})
assert len(query.get("matches", [])) > 0

server.terminate()
print("Integration tests passed")

Why This Works

*1. Startup order matters. The MCP server initializes in a specific sequence: open the database, load the failure cache into memory, register tool handlers, then start listening on STDIO. The integration test caught a bug where the cache was queried before it finished loading - returning stale data from the previous session.

2. JSON-RPC framing is not trivial. A 4KB failure report with URL-encoded characters caused readlines() to split mid-payload. Only a real STDIO conversation reveals this.

3. State accumulates. After 150 entries, deduplication used LIMIT 1without ORDER BY, returning a random match. Unit tests with 3-5 entries never caught this.

Gotchas

*Don't test in production. My first test locked the live database for 12 seconds. Use tempfile.mkstemp() and clean up in finally.

Timeouts. The server waited for a signal that never came on STDIO mode. Add a 3-second alarm.

Process cleanup. Always terminate and wait in try/finally.

What About You?

When was the last time an integration test caught something your unit tests missed?