I Added a "Failure Library" to My MCP Server. Now My AI Agents Warn Each Other About Crashes Before They Happen.

Hook

My MCP server kept crashing in production. Every time it restarted, my AI agents would blindly retry the same failing operations, wasting tokens and time. I needed a way for them to learn from failures — and share those lessons with each other.

So I built a failure library. Now when an agent hits an error, it logs it. The next agent checks the library first and either avoids the trap entirely or applies a verified fix.

The Fix

Here's the core pattern — a simple JSON-based failure cache that any MCP server can adopt:

import json
import time
from pathlib import Path
from typing import Optional

FAILURE_LIBRARY = Path.home() / ".mcp" / "failures.json"

def log_failure(task: str, error: str, attempted_fix: str, result: str):
    """Log a failure to the shared library."""
    entry = {
        "task": task,
        "error": error,
        "attempted_fix": attempted_fix,
        "result": result,
        "timestamp": time.time(),
    }
    failures = []
    if FAILURE_LIBRARY.exists():
        failures = json.loads(FAILURE_LIBRARY.read_text())
    failures.append(entry)
    FAILURE_LIBRARY.parent.mkdir(parents=True, exist_ok=True)
    FAILURE_LIBRARY.write_text(json.dumps(failures, indent=2))

def check_failure_library(task: str) -> Optional[dict]:
    """Check if this task has a known failure with a verified fix."""
    if not FAILURE_LIBRARY.exists():
        return None
    failures = json.loads(FAILURE_LIBRARY.read_text())
    for f in reversed(failures):  # most recent first
        if f["task"] == task and f["result"] == "fixed":
            return f
    return None

Now wrap your MCP tool calls with a pre-check:

# Before calling a potentially flaky tool
known = check_failure_library("browser_navigate")
if known:
    print(f"⚠ Known issue: {known['error']}")
    print(f"✅ Verified fix: {known['attempted_fix']}")
    # Apply the fix instead of retrying blindly
else:
    # First time — proceed normally, log if it fails
    try:
        result = page.goto(url)
    except Exception as e:
        log_failure("browser_navigate", str(e), "add retry with 5s delay", "fixed")

Why This Works

The failure library works because it turns individual agent mistakes into collective knowledge. Three things make it effective:

1. Reverse lookup — checking most-recent failures first means the latest fix is always tried first. This matters because the same operation can fail for different reasons over time. A newer fix might address a root cause that an older entry only papered over.

2. Result tagging — "fixed" vs "failed" entries let agents distinguish verified solutions from dead ends. This is critical because not every attempted fix actually works. An agent might try adding a retry loop, discover it doesn't help, and log "result": "failed". The next agent sees that and skips straight to a different approach.

3. Shared filesystem — any agent on the same machine reads the same library, so lessons propagate instantly. No network calls, no database connections, no coordination overhead. The file is the message.

The JSON format is intentionally simple. No database, no network calls — just a file that any language can read and write. This means your Python agent, your Node.js MCP server, and your Rust CLI tool can all share the same failure library without any special integration.

Putting It All Together

Here's how I integrated the failure library into my actual MCP server's tool handler:

class MCPToolHandler:
    def __init__(self):
        self.failure_lib = FailureLibrary()

    async def execute_tool(self, tool_name: str, args: dict):
        # Check failure library before executing
        known_failure = self.failure_lib.check(tool_name)
        if known_failure:
            # Apply the verified fix automatically
            args = self._apply_fix(tool_name, args, known_failure)

        try:
            result = await self._call_tool(tool_name, args)
            return result
        except Exception as e:
            # Log the failure for future agents
            self.failure_lib.log(
                task=tool_name,
                error=str(e),
                attempted_fix=self._suggest_fix(tool_name, e),
                result="pending"  # will be updated after retry
            )
            # Try the fix
            fixed_args = self._apply_fix(tool_name, args, known_failure)
            try:
                result = await self._call_tool(tool_name, fixed_args)
                self.failure_lib.update_result(tool_name, "fixed")
                return result
            except Exception as e2:
                self.failure_lib.update_result(tool_name, "failed")
                raise

The key insight is that the failure library becomes part of your tool's retry logic. Instead of a dumb retry loop, each retry consults the library and applies a different strategy.

Gotchas

• Concurrent writes: if multiple agents log failures simultaneously, you can get race conditions. Use file locking (fcntl on Unix, msvcrt on Windows) for production use. Here's a minimal lock wrapper:

import fcntl
def safe_write(path, data):
    with open(path, 'w') as f:
        fcntl.flock(f.fileno(), fcntl.LOCK_EX)
        f.write(data)
        fcntl.flock(f.fileno(), fcntl.LOCK_UN)

• Library bloat: failures accumulate over time. Add a TTL (e.g., 30 days) and prune old entries. I run a cleanup cron job every Sunday:

# Prune failures older than 30 days
python -c "
import json, time
from pathlib import Path
lib = Path.home() / '.mcp' / 'failures.json'
if lib.exists():
    failures = json.loads(lib.read_text())
    cutoff = time.time() - 30 * 86400
    fresh = [f for f in failures if f['timestamp'] > cutoff]
    lib.write_text(json.dumps(fresh, indent=2))
    print(f'Pruned {len(failures) - len(fresh)} old entries')
"

• False positives: an entry marked "fixed" might not work in a different context. Always include the environment details (OS, library version) in the task description. I use a composite key: "browser_navigate|linux|playwright-1.40".

• Network tools: the library only works for local failures. For API errors, you need a centralized store (Redis, SQLite). I sync local failures to a Redis instance every 5 minutes for cross-machine sharing.

What about you?

Have you built something similar — a way for your AI agents to share knowledge about failures? Or do you rely on a different pattern for error recovery? I'm curious whether the JSON-file approach scales to larger agent fleets or if a proper database is needed from day one.

Drop a comment below — I read every response.

Comments

[Mads Hansen]

The useful boundary is “verified remediation,” not merely shared failure text. I’d key entries by tool version, config digest, input shape, and typed error class, and only mark a fix verified after the original reproducer passes. Store a reviewed remediation ID that maps to code rather than auto-applying arbitrary strings from the library. Also track last success and invalidate entries on version changes. For the JSON implementation, lock the entire read-modify-write and use atomic rename; locking only the final write can still lose concurrent updates.

[Reid Marlow]

The useful bit is separating the failure note from the permission to use it. I’d treat the library as a set of candidate remediations, keyed by tool version, config digest, input shape, and typed error class. Only promote one to “verified” after the original reproducer passes again.Otherwise the library can become a very patient way to spread a bad fix across every agent in the fleet. Same bug, now with persistence.