Four Traps in MCP Health Checking: What Broke My Overnight Batches

An MCP server dying at 2 a.m. used to mean waking up to a log full of connection refused and zero work done. Building a health-check hook that records failures outside the context window fixed that — my overnight batches haven't been killed by an MCP outage since.

Why This Mechanism Works

What Happened When an MCP Server Went Down

When I first started automating with Claude Code, there was one failure mode I hated more than any other: a batch job kicked off at night, stopped dead by an MCP server timeout. I'd wake up to a pile of connection refused logs and nothing having advanced since the previous evening. For SNS post automation that's survivable, but the time it stopped a client deliverable generation run, I genuinely panicked.

MCP is the mechanism that lets Claude Code use browser operations, DB lookups, external API calls, and so on as tools. From Claude's side it just looks like calling a tool named something like mcp__obsidian__search, but underneath, communication with a local process or a remote HTTP server is running. When that server stops responding for whatever reason, Claude Code returns the tool call as an error and the whole flow of the session jams up.

The problem wasn't only "why did it stop." There were cases where it was stopped by a 429 (rate limit), yet it would re-call 30 seconds later → another 429 → stopped again, looping forever. There were also cases where a 503 (service temporarily unavailable) was judged the same as a 401 (expired auth), pointlessly running a re-authentication flow. Unless you vary the strategy per status code, it doesn't matter how fast a model you use — it's wasted.

The Essence of Fixing the "Environment" Instead of the "Work"

Maintaining an autonomous environment at ¥1.2M/month revenue, what I noticed is that the time spent on "mechanisms that keep things from stopping" has a higher long-term ROI than the code added to increase earnings.

Back at ¥600K/month I thought "more tasks means more earnings." After being laid off and dropping to zero, my thinking changed while rebuilding it from scratch. Adding tasks doesn't help if the environment is unstable — throughput pins to a ceiling. Conversely, killing a single infrastructure-level problem raises the completion rate across all existing tasks.

The MCP health check is the classic example of this. By wiring ~/.claude/scripts/hooks/mcp-health-check.js into a hook, an HTTP probe runs before Claude Code calls a tool, and depending on the response status it dispatches to "block immediately," "retry after backoff," or "run a reconnect command and re-probe." The verdict is persisted to ~/.claude/mcp-health-cache.json, so even when context is compacted, the health record carries over.

Why the Cache Counts as a "Context Compaction Countermeasure"

When a Claude Code session runs for a long time, the conversation history gets compacted. Even if there was information in the past saying "this server was down," that doesn't survive into the post-compaction context. The result is waste: repeatedly attempting tool calls against a server already known to be unhealthy, and receiving an error each time.

A file-based cache is independent of the context. ~/.claude/mcp-health-cache.json doesn't disappear no matter how much the session is compacted. When the health-check hook runs on the next turn, it loads the previous state from the file, and until nextRetryAt has passed it blocks immediately without even re-probing. The idea of holding state outside the context window is what matters fundamentally.

Common Misconceptions

Some people think, "why not just do error handling in Claude's prompt?" I actually tried it. A system prompt saying "if this tool errors, try another approach" works reasonably well for a one-off error. But in a situation where the MCP server is down and failures happen back to back, the model burns a huge number of tokens on "trying." And on the next turn, it goes right back to calling the same server. Writing the fact that "the server is dead" outside the context and blocking at the hook level is overwhelmingly cleaner.

Another common misconception is "MCP's own retry settings are enough." The MCP protocol has transport-layer retries, but it has no feature for reading status codes and switching strategy. 429 and 503 call for different backoff durations, and 401/403 are cases that need re-authentication rather than a retry. Implementing this dispatch at the application layer is what this hook is for.

The Overall Flow

When the Hook Intervenes

mcp-health-check.js responds to two kinds of Claude Code hook events. It's written verbatim in the comment at the top of the code (lines 7–12).

- PreToolUse: probe MCP server health before MCP tool execution
- PostToolUseFailure: mark unhealthy servers, attempt reconnect, and re-probe

PreToolUse is called before the tool is executed. Here it fires a probe, checks whether the server is alive, and decides whether to allow execution or block with exit code 2. PostToolUseFailure is called after a tool returns an error. It parses the error text to identify the failure pattern, marks the server unhealthy, and attempts a reconnect.

Full Flow Diagram

Claude Code が mcp__* ツールを呼ぶ
        │
        ▼ PreToolUse フック起動
┌──────────────────────────────────────────────────────┐
│            mcp-health-check.js                       │
│                                                      │
│  ① mcp-health-cache.json を読む                      │
│     status=healthy かつ expiresAt が未来？            │
│     YES ─────────────────────────────────────────→  │ exit 0
│     NO  ↓                                            │ （ツール実行へ）
│                                                      │
│  ② nextRetryAt が未来（unhealthy クールダウン中）？   │
│     YES → ブロック ──────────────────────────────→  │ exit 2
│     NO  ↓                                            │ （ツールをスキップ）
│                                                      │
│  ③ プローブ実行                                       │
│     HTTPサーバー → GET リクエスト（5秒タイムアウト）  │
│     stdioサーバー → プロセス起動（5秒生存確認）       │
│                                                      │
│     レスポンスのステータスコード判定                   │
│     ┌──────────────────────────────────────────┐    │
│     │ ECONNREFUSED / ENOTFOUND / タイムアウト   │──→ │
│     │  → 即 markUnhealthy & exit 2             │    │
│     ├──────────────────────────────────────────┤    │
│     │ 401 / 403 / 429 / 503                    │──→ │
│     │  → reconnect コマンドを実行               │    │
│     │  → 成功すれば再プローブ                   │    │
│     │  → 再プローブ OK → markHealthy & exit 0  │    │
│     │  → 再プローブ NG → markUnhealthy & exit 2│    │
│     ├──────────────────────────────────────────┤    │
│     │ 200 系 / 400 / 401 / 403 / 405 / 406    │    │
│     │（"到達できた"証明として healthy 扱い）    │──→ │ exit 0
│     └──────────────────────────────────────────┘    │
│                                                      │
│  ④ 状態を mcp-health-cache.json に書き出す            │
└──────────────────────────────────────────────────────┘
        │
        ▼ ツール実行後にエラーが出た場合
┌──────────────────────────────────────────────────────┐
│ PostToolUseFailure フック                             │
│   エラーテキストを FAILURE_PATTERNS と照合            │
│   failureCode 特定 → markUnhealthy → reconnect試行  │
│   → 再プローブ OK なら markHealthy                   │
└──────────────────────────────────────────────────────┘

What the Constants Say About the Design

Reading the actual code, the design intent shows up in the numbers (lines 22–26).

const DEFAULT_TTL_MS    = 2 * 60 * 1000;   // 2分
const DEFAULT_TIMEOUT_MS = 5000;            // 5秒
const DEFAULT_BACKOFF_MS = 30 * 1000;       // 30秒（初回バックオフ）
const MAX_BACKOFF_MS     = 10 * 60 * 1000; // 10分（上限）

A TTL of 2 minutes is the trade-off between "I don't want to run a probe every single time" and "I don't want to hold a stale result too long." In scenarios where Claude Code calls tools back to back, hitting the same server multiple times within 2 minutes is not unusual. With a cache, there's no need to run an HTTP probe each time, and latency drops.

The 5-second timeout is the threshold for confirming that a local stdio server "can start as a process." Rather than startup completion, it judges normality by the process "staying alive for 5 seconds" (the timer logic in probeCommandServer, lines 428–472). Even for a heavy server that takes more than 5 seconds to start, this can be adjusted with the environment variable ECC_MCP_HEALTH_TIMEOUT_MS.

The Exponential Backoff Formula

The backoff calculation in the markUnhealthy function (lines 213–229) is compressed into one line.

const nextRetryDelay = Math.min(
  backoffBase * (2 ** Math.max(failureCount - 1, 0)),
  MAX_BACKOFF_MS
);

backoffBase defaults to 30 seconds. When failureCount is 1, 2 ** 0 = 1 gives 30 seconds; the second time 2 ** 1 = 2 gives 60 seconds; the third 120 seconds; the fourth 240 seconds, doubling each time, capping out at a maximum of 600 seconds (10 minutes). It waits 30 seconds after the first failure, and if it hasn't recovered there, widens the interval to 1 minute, 2 minutes, 4 minutes — which avoids the situation of endlessly firing pointless probes at a downed server.

The Status Code Dispatch Logic

The definition of HEALTHY_HTTP_CODES (line 32) looks odd at first glance.

const HEALTHY_HTTP_CODES = new Set([
  200, 201, 202, 204,
  301, 302, 303, 304, 307, 308,
  400, 401, 403, 405, 406
]);

Some 400-level codes are treated as "healthy." The reason is written in the code's comment (lines 29–32).

// The preflight HTTP probe only checks reachability; it does not have access to
// Claude Code's stored OAuth bearer token. Treat auth-gated responses as
// reachable so the real MCP client can attempt the authenticated call. A
// Streamable HTTP MCP server can also return 406 to a bare GET that omits
// Accept: text/event-stream; that still proves the endpoint is alive.

The preflight probe is a GET request with no OAuth token. An endpoint that requires authentication returns 401 or 403, but that means it is "correctly rejecting an unauthenticated request" — evidence that the server is alive. 406 is a normal rejection response to a request without the Accept: text/event-stream header. These are all codes that prove "it was reachable."

Meanwhile, the set targeted for reconnect is a subset (line 33).

const RECONNECT_STATUS_CODES = new Set([401, 403, 429, 503]);

401 and 403 are treated as "reachable" in the probe context, but when they come back after an actual tool call, the meaning is different. When the detectFailureCode function is called in PostToolUseFailure, a 401/403 in the error message is taken as "authentication failure" and becomes the trigger for a reconnect. The key point is that the probe verdict and the post-failure verdict operate at different layers.

FAILURE_PATTERNS and Text Analysis

MCP errors don't necessarily come back as HTTP status codes. Errors from stdio servers come back as text. FAILURE_PATTERNS (lines 34–40) is the mechanism that identifies the failure type from that text via regular expressions.

const FAILURE_PATTERNS = [
  { code: 401, pattern: /\b401\b|unauthori[sz]ed|auth(?:entication)?\s+(?:failed|expired|invalid)/i },
  { code: 403, pattern: /\b403\b|forbidden|permission denied/i },
  { code: 429, pattern: /\b429\b|rate limit|too many requests/i },
  { code: 503, pattern: /\b503\b|service unavailable|overloaded|temporarily unavailable/i },
  { code: 'transport', pattern: /ECONNREFUSED|ENOTFOUND|EAI_AGAIN|timed? out|socket hang up|connection (?:failed|lost|reset|closed)/i }
];

From error messages containing strings like "unauthorized," "auth expired," "rate limit," or "ECONNREFUSED," it automatically identifies the failure code. The transport code isn't an HTTP status — it's the case where the network connection itself is cut. This is a state of "can't even reach the server," where backoff takes priority over reconnect.

The Structure of the Cache File

~/.claude/mcp-health-cache.json is written out by the saveState function (lines 95–102), formatted with JSON.stringify. The actual file has a structure like this.

{
  "version": 1,
  "servers": {
    "obsidian": {
      "status": "healthy",
      "checkedAt": 1753666200000,
      "expiresAt": 1753666320000,
      "failureCount": 0,
      "lastError": null,
      "lastFailureCode": null,
      "nextRetryAt": 1753666200000,
      "lastRestoredAt": 1753666200000,
      "source": "~/.claude/settings.json"
    },
    "agentmemory": {
      "status": "unhealthy",
      "checkedAt": 1753665900000,
      "expiresAt": 1753665900000,
      "failureCount": 3,
      "lastError": "ECONNREFUSED 127.0.0.1:3001",
      "lastFailureCode": "transport",
      "nextRetryAt": 1753666020000,
      "lastRestoredAt": null
    }
  }
}

It's a two-stage gate: if expiresAt is in the future, pass without re-probing; if nextRetryAt is in the future, block without even retrying. The more failureCount accumulates, the longer the backoff stretches, so a repeatedly-down server gets progressively wider attempt intervals. lastRestoredAt is a record of "when it recovered," usable for after-the-fact uptime analysis.

The source field is the path to the config file. The configPaths function (lines 54–72) searches in the order: the current directory's .claude.json → the current directory's .claude/settings.json → home's .claude.json → home's .claude/settings.json, and the first one found is used. Even when you have per-project MCP configuration, the correct config is referenced.

How to Configure the Reconnect Command

The reconnectCommand function (lines 516–527) reads the command from environment variables.

const key = `ECC_MCP_RECONNECT_${String(serverName).toUpperCase().replace(/[^A-Z0-9]/g, '_')}`;
const command = process.env[key] || process.env.ECC_MCP_RECONNECT_COMMAND || '';

If the server name is agentmemory, it looks for an environment variable named ECC_MCP_RECONNECT_AGENTMEMORY. If not found, it uses the global fallback ECC_MCP_RECONNECT_COMMAND. If the command string contains {server}, it is expanded to the server name (lines 524–526).

For example, if you use PM2 as your process manager, you can configure it like this.

export ECC_MCP_RECONNECT_COMMAND="pm2 restart {server}"

With this, when the agentmemory server returns a 401 or 503, pm2 restart agentmemory runs automatically, and tool execution is allowed only after a re-probe confirms it's normal. When a reconnect succeeds, reconnect-command is recorded in markHealthy's restoredBy field (lines 607–609).

At this point you should have a grasp of the hook's overall shape. In the next section I'll dig into the actual stumbling points and the traps that are easy to fall into during configuration.

Implementation Details

extractMcpTarget — Decomposing the Server Name from the Tool Name

The first thing the hook does is identify "which MCP server is this call for." The extractMcpTarget function (lines 133–167) handles that.

if (!toolName.startsWith('mcp__')) {
  return null;
}

const segments = toolName.slice(5).split('__');
if (segments.length < 2 || !segments[0]) {
  return null;
}

return {
  server: segments[0],
  tool: segments.slice(1).join('__')
};

Given a name like mcp__obsidian__search_notes, slice(5) makes it obsidian__search_notes, and split('__') makes it ['obsidian', 'search', 'notes']. segments[0] is the server name, and the rest joined with double underscores is the tool name.

That said, parsing the tool name is strictly a fallback. It looks for an explicit server field first (lines 135–145).

const explicitServer = input.server
  || input.mcp_server
  || input.tool_input?.server
  || input.tool_input?.mcp_server
  || input.tool_input?.connector
  || null;

The reason it walks multiple paths is that the hook event's schema can have fields in shifted positions depending on the Claude Code version and connection method (HTTP/stdio). Furthermore, if JSON parsing fails and the truncated flag is set, extractMcpTargetFromRaw (lines 169–179) applies a regular expression to the raw string to extract the same information. It's a design that doesn't give up even when parsing fails.

The Two-Stage Gate in handlePreToolUse

handlePreToolUse (lines 567–630) looks simple, but two cache checks run in succession.

// 第1ゲート: healthy かつキャッシュ有効 → プローブなしで通過
if (previous.status === 'healthy' && Number(previous.expiresAt || 0) > now) {
  return { rawInput, exitCode: 0, logs };
}

// 第2ゲート: unhealthy かつクールダウン中 → プローブなしでブロック
if (previous.status === 'unhealthy' && Number(previous.nextRetryAt || 0) > now) {
  logs.push(
    `[MCPHealthCheck] ${target.server} is marked unhealthy until ${new Date(previous.nextRetryAt).toISOString()}; skipping ${target.tool || 'tool'}`
  );
  return { rawInput, exitCode: shouldFailOpen() ? 0 : 2, logs };
}

"healthy and within TTL" passes, "unhealthy and in backoff" is blocked, and only in neither case (unhealthy but backoff expired, or the cache is empty) is a probe executed. In sessions that call tools at high frequency, almost all calls pass at the first gate, so HTTP request latency is effectively zero.

The conditions that trigger a reconnect need similar care (lines 601–603).

let reconnect = { attempted: false, success: false, reason: 'probe failed' };
if (probe.failureCode || previous.status === 'unhealthy') {
  reconnect = attemptReconnect(target.server);

A reconnect doesn't run just because "the probe failed." It requires either probe.failureCode (a status code for which reconnect is enabled) or the condition "already unhealthy since last time." Running a reconnect command every time when you can't even connect due to ECONNREFUSED is wasteful, but when the server exists yet is unusable — as with 429 or 503 — actively attempting to reconnect is worthwhile; that judgment is embedded here.

probeCommandServer — The "5-Second Liveness Check" for stdio Servers

For an HTTP server, firing a GET with requestHttp tells you what you need. How do you check a stdio server? The approach in probeCommandServer (lines 301–481) is distinctive.

Actually start the process and watch it for 5 seconds. If it stays alive, it's healthy; if it exits before then, it's not.

timer = setTimeout(() => {
  // タイムアウト到達 = 5秒間プロセスが生存 → 正常

  // ただし: ロードされたマシンではexitイベントがタイマーより
  // わずかに遅れて届くことがある。高速クラッシュを見逃さないため
  // プロセスの状態を再確認する
  if (child.exitCode !== null || child.signalCode !== null) {
    attemptFinish({
      ok: false,
      statusCode: child.exitCode,
      reason: stderr.trim() || `process exited before handshake (...)`
    });
    return;
  }

  // SIGTERMで終了させ、200ms後にSIGKILLで確実に始末する
  child.kill('SIGTERM');
  setTimeout(() => {
    try { child.kill('SIGKILL'); } catch { /* ignore */ }
  }, 200).unref?.();

  attemptFinish({
    ok: true,
    statusCode: null,
    reason: `${serverName} accepted a new stdio process`
  });
}, timeoutMs);

As the comment says, the "fast-crashing server" problem really is a nuisance. If the process exits right after startup, the exit event and the timer callback can arrive at nearly the same moment. If the timer runs first, it would return ok: true without a child.exitCode check. As a countermeasure, the timer callback re-checks child.exitCode !== null, and if it's already dead, drops it to ok: false.

Windows behavior is worked out too (lines 329–334, 438–461). A command without an extension, like npx, needs to resolve to npx.cmd on Windows, and on top of that, from Node 18.20 onward, executing .cmd via a shell was restricted as part of the CVE-2024-27980 fix. The code lines up fallbacks of command.cmd / command.exe / command.bat in a candidates array, and tries the next candidate in order on ENOENT. It also executes .cmd/.bat files via a shell, but has a safety valve that refuses shell execution if the command string contains shell metacharacters (&|<>^%() etc.) (line 339, UNSAFE_SHELL_CHARS).

failureSummary — Collecting Error Text From Every Direction

failureSummary (lines 231–244) is quietly important.

const pieces = [
  typeof input.error === 'string' ? input.error : '',
  typeof input.message === 'string' ? input.message : '',
  typeof input.tool_response === 'string' ? input.tool_response : '',
  typeof output === 'string' ? output : '',
  typeof output?.output === 'string' ? output.output : '',
  typeof output?.stderr === 'string' ? output.stderr : '',
  typeof input.tool_input?.error === 'string' ? input.tool_input.error : ''
].filter(Boolean);

return pieces.join('\n');

Which field the MCP error text lands in differs by server implementation and Claude Code version. The case where it's in error, the case where it's in tool_response, the case where it's in output.stderr — I've encountered all of them in practice. By passing the string joined from all fields with pieces.join('\n') to detectFailureCode, the regular expressions can catch the error text no matter which field it's in.

fail-open and the PostToolUseFailure exit code

shouldFailOpen (lines 557–559) is a one-liner over an environment variable.

function shouldFailOpen() {
  return /^(1|true|yes)$/i.test(String(process.env.ECC_MCP_HEALTH_FAIL_OPEN || ''));
}

It's the setting that lets through tool calls to unhealthy servers. You use it during development when you want to "have the hook running but not blocking, just to observe." In PreToolUse, it returns exit 0 (pass) instead of exit 2 (block).

The exitCode for PostToolUseFailure is always 0 (line 677). This is an important design point: the post-failure hook's purpose is recording state, not blocking tool execution. Changing the exit code of a tool call that has already failed is meaningless; it only writes logs and updates the cache, handing the state over to the next PreToolUse.

---

Where I Got Stuck

From wiring this mechanism into my own environment to having it stable, I got stuck in three situations. I'll write the symptom, the cause, and the fix in order.

Stuck #1: A stdio Server That "Could Start" but Errored Every Time

Symptom: The agentmemory server remained in the cache with status: healthy, yet actual tool calls returned ECONNREFUSED. The health check looked meaningless.

Cause: A stdio server probe only confirms that "the process stays alive for 5 seconds." The process starts, gets terminated with SIGTERM after 5 seconds, and then when the real tool call arrives, the server starts again from zero. But in a real startup, it takes time to complete the MCP handshake (JSON message exchange over stdio). A different error was occurring during that handshake.

Specifically, a server that tried to start without the environment variable ANTHROPIC_API_KEY being set was crashing right after startup. At probe time it stayed alive for 5 seconds so it got recorded as healthy, but during the actual handshake it would crash a few seconds in.

Fix: Check first whether the server process's environment variables are all in place. mcp-health-check.js merges config.env into the process's environment variables when starting (lines 306–309). Confirming that the same environment variables as the production startup are in the config file's env section is the first order of business. Only after understanding that "healthy in the probe" and "healthy in the tool call" are strictly different verdicts did the debugging direction finally become clear.

Stuck #2: 503s Without Reconnect Piled Up and the Server Was Unusable for a Long Time

Symptom: The Obsidian remote server temporarily returned 503s overnight. Checking the next morning, the server itself had long since recovered, but nextRetryAt in mcp-health-cache.json pointed to a future time and it stayed blocked.

Cause: Because 503 is included in RECONNECT_STATUS_CODES, a reconnect is attempted when a 503 comes in. But I had set neither ECC_MCP_RECONNECT_OBSIDIAN nor ECC_MCP_RECONNECT_COMMAND. Without a reconnect command, attemptReconnect returns attempted: false (line 531), only markUnhealthy runs, and failureCount accumulates. Three 503s make failureCount=3, and backoff is 30 * (2 ** 2) = 120 seconds. Five more over the next 30 minutes makes failureCount=5 and 30 * (2 ** 4) = 480 seconds. As a dozen-plus errors piled up overnight, the backoff was pinned at the ceiling of 600 seconds (10 minutes). Even after the server recovered, a probe only ran every 10 minutes, and all tool calls in between were blocked.

Fix: Either configure a reconnect command, or set ECC_MCP_HEALTH_BACKOFF_MS to a small value (e.g. 10000 = 10 seconds) to lower the backoff base. I set ECC_MCP_RECONNECT_COMMAND=echo noop (a do-nothing dummy) in my launchd plist, which gets it treated as "reconnect attempted" and suppresses failureCount accumulation. The proper approach is to manage the service with PM2 or systemd and configure a real reconnect command.

export ECC_MCP_RECONNECT_OBSIDIAN="pm2 restart obsidian-mcp"

Once this setting was in, the cycle started working: a 503 comes in → PM2 restarts → a re-probe confirms normal → immediate return to healthy.

Stuck #3: A Project Directory's settings.json Hid the Home Configuration

Symptom: When starting Claude Code in a particular project directory, the MCP health check logged "No MCP config found for obsidian" and skipped the probe, yet the tool call itself ran. The hook appeared to be functioning while not actually functioning.

Cause: The configPaths function (lines 54–72) searches the current directory's config file first.

return [
  path.join(cwd, '.claude.json'),
  path.join(cwd, '.claude', 'settings.json'),   // ← これが先に見つかる
  path.join(home, '.claude.json'),
  path.join(home, '.claude', 'settings.json')
];

That project had a .claude/settings.json, and it defined only project-specific MCP servers (like playwright). resolveServerConfig('obsidian') doesn't stop at the point of finding the first file in list order — if that file doesn't have obsidian, it moves on to the next file. In readJsonFile's loop, if data?.mcpServers?.[serverName] is null it returns null and continues the loop — but what actually caused the problem was a case where I had pointed the environment variable ECC_MCP_CONFIG_PATH at the home config file and there was a typo in that path.

# 誤（タイポあり）
export ECC_MCP_CONFIG_PATH="/Users/~/.claude/settings.json"

# 正
export ECC_MCP_CONFIG_PATH="${HOME}/.claude/settings.json"

path.resolve returned the nonexistent path /Users/~/.claude/settings.json, readJsonFile returned null, and it was treated as config not found. Because setting ECC_MCP_CONFIG_PATH completely overrides the normal search paths (lines 55–61), a typo makes all server configuration disappear.

Fix: Use an absolute path for ECC_MCP_CONFIG_PATH. Tilde expansion is something the shell does; Node's path.resolve does not do it. When writing it in a launchd plist, ~ is not expanded, so you need to write it as /Users/your-home-directory, ${HOME} (via a shell script), or the equivalent of os.homedir().

<!-- launchd plist では ~ が展開されない -->
<key>ECC_MCP_CONFIG_PATH</key>
<string>/Users/lily/.claude/settings.json</string>

Writing the actual home directory path literally is the most reliable.

Stuck #4: I Left ECC_MCP_HEALTH_FAIL_OPEN in the Production Environment

Symptom: The hook was working correctly and unhealthy was recorded in the cache. Yet tool calls went through and errors flowed into Claude's context. It produced the token waste of receiving the same error 20 times in one session.

Cause: I had forgotten to remove ECC_MCP_HEALTH_FAIL_OPEN=1, which I'd set during debugging, from my launchd plist. As long as this setting exists, shouldFailOpen() returns true and all tool calls to unhealthy servers become exit 0 (pass). The hook is running but blocking nothing.

So there was an inconsistency where looking at mcp-health-cache.json showed status: unhealthy recorded, and yet tool calls went through.

Fix: Treat ECC_MCP_HEALTH_FAIL_OPEN as a debug-only flag and never write it in a permanent launchd plist. I made myself an operational rule to delete it as soon as debugging is over. To check whether the hook is blocking as intended, the quickest route is to read ~/.claude/mcp-health-cache.json directly and, in a state where a server with status: unhealthy exists, confirm that the skipping wording appears in the hook's stderr log when a tool call runs.

# フックのログをリアルタイムで確認（stderrはClaude Codeのhookログに流れる）
tail -f ~/.claude/logs/hooks.log | grep MCPHealthCheck

After getting past these four sticking points, the environment stabilized. Now it's normal for a batch running overnight to have finished 100 items by the time I wake up. Since introducing this mechanism, a batch has never once been stopped by an MCP server going down.

Pitfalls

The previous section covered four sticking points in detail. Here I'll enumerate, as a list, the traps I actually hit or that are easy to overlook. Ones that overlap with P2 are omitted.

• When CLAUDE_HOOK_EVENT_NAME isn't passed, all calls are processed as PreToolUse

Line 704 of the code has const eventName = process.env.CLAUDE_HOOK_EVENT_NAME || 'PreToolUse';. If this environment variable is missing, event determination is pinned to PreToolUse. There are cases where a hook you meant to register as PostToolUseFailure was actually running as PreToolUse. The first step in isolating this is to dump process.env to stderr and check whether Claude Code is passing ENV correctly.

• When stdin exceeds 1MB, the truncated flag is set, and unless fail-open is on it becomes an immediate block

MAX_STDIN = 1024 * 1024 (line 22) is that limit. If, on a large tool call that stuffs an entire file's contents into tool_input, the hook input exceeds this value, it tries to identify the target server with the parse still incomplete. If the target can be identified, you can raise the limit with ECC_HOOK_INPUT_MAX_BYTES, but if it can't be parsed completely, it's blocked with exit 2 unless fail-open is on (lines 695–701). Without knowing this behavior, it looks like the mysterious phenomenon of "the hook falsely blocks only certain tools."

• saveState is designed to swallow errors, so write failures continue silently

saveState at lines 95–102 catches all errors with try/catch and, as the comment says — "Never block the hook on state persistence errors." — silently continues. Disk full, insufficient permissions, the ~/.claude/ directory being gone: in every case the hook keeps returning exit 0 or exit 2. But because the cache isn't written out, healthy verdicts disappear after the TTL and a probe runs every time. If you feel like "for some reason an HTTP probe runs every time," start with a disk check via df -h ~/.claude/ and a permission check via ls -la ~/.claude/mcp-health-cache.json.

• You can't tell which file resolveServerConfig is using without checking the source field

resolveServerConfig (lines 181–197) scans all files in configPaths() order and, at the point the server name is found, records that path in the source field. If a server of the same name is defined both in a project-specific .claude/settings.json and in home's ~/.claude/settings.json, the project side wins. You can check whether an unintended config is being used via the source field in ~/.claude/mcp-health-cache.json. 80% of "I fixed the home config but it isn't reflected" symptoms are this.

• The reconnect command is executed with shell:true, so you need to be careful about the structure of the command string

attemptReconnect (lines 528–555) executes with spawnSync(command, { shell: true, ... }). Command chains containing && or ; work fine, but command substitutions like $(date) can get expanded unintentionally. Keeping the value of ECC_MCP_RECONNECT_COMMAND to a simple single command like pm2 restart {server} is safest. Using the {server} placeholder (lines 524–526) saves you from having to set a separate environment variable per server.

• failureCount isn't reset until markHealthy is called, so the backoff stays pinned at the ceiling and never recovers

markUnhealthy accumulates previous.failureCount + 1 every time (line 215). Meanwhile, markHealthy force-resets to failureCount: 0 (lines 200–210). The problem is that when no reconnect command is configured and the backoff pins at the ceiling (600 seconds), you're made to wait 10 minutes until the next re-probe. All tool calls in that window are blocked. Even after a long-unstable server recovers, recovery is only confirmed every 10 minutes. The fastest way to clear this state is to manually delete the target server's entry from the cache file.

  python3 -c "
  import json, pathlib
  p = pathlib.Path.home() / '.claude/mcp-health-cache.json'
  s = json.loads(p.read_text())
  s['servers'].pop('agentmemory', None)
  p.write_text(json.dumps(s, indent=2))
  print('reset done')
  "

• Environment variables missing from a stdio server's config.env are the cause of "it doesn't work when started from launchd"

probeCommandServer (lines 306–309) merges config.env into process.env at process startup. In an interactive zsh session, ANTHROPIC_API_KEY is automatically inherited from .zshrc, but Claude started from launchd doesn't have the login shell's environment variables. Most cases of the symptom "works when I try it by hand, fails in the overnight batch" have their cause here. The fundamental countermeasure is to explicitly enumerate the required API keys in every server's config.env section.

• There are cases where a probe to an HTTP server keeps returning 429 and enters a reconnect loop

Because 429 is included in RECONNECT_STATUS_CODES, a reconnect runs when the probe returns 429 (line 33). Even if the reconnect command executes, if the underlying problem (rate limiting) isn't resolved, the re-probe also returns 429, markUnhealthy runs, and failureCount increases. It's a vicious circle where the probe itself consumes the rate limit. To prevent this, you can either set ECC_MCP_HEALTH_TTL_MS longer to reduce probe frequency, or change the target server's probe URL to a health endpoint that doesn't require authentication.

• Assuming that blocking works even though the PostToolUseFailure hook's exit code is 0

The return value of handlePostToolUseFailure is always exitCode: 0 (line 677). This is correct by design — returning a block for a "tool call that has already failed" is too late. This hook's job is "writing state so the next PreToolUse blocks," and the exit code has no meaning. Most reports of "the PostToolUseFailure hook is running but nothing gets blocked" are resolved by checking whether the cache is being read correctly on the next PreToolUse.

---

Best Practices

Over six months of supporting a ¥1.2M/month autonomous environment with this hook, here are 12 things I stick to.

1. Always configure a reconnect command, even a dummy one

When RECONNECT_STATUS_CODES errors pile up without a reconnect command, failureCount snowballs and the backoff pins at the ceiling. Access to an already-recovered server stays blocked for a long time. Under PM2 management, configure a real one; otherwise even just echo noop gets it treated as "reconnect attempted" and suppresses failureCount accumulation.

# PM2管理下なら本物を
export ECC_MCP_RECONNECT_COMMAND="pm2 restart {server}"

# とりあえず蓄積を止めるだけなら
export ECC_MCP_RECONNECT_COMMAND="echo noop"

2. Don't write ECC_MCP_HEALTH_FAIL_OPEN in your launchd plist

While this variable is set, shouldFailOpen() (lines 557–559) returns true and all tool calls to unhealthy servers become exit 0 (pass). The hook looks like it's working while blocking nothing. Treat it as a debug-only flag and make an operational rule to remove it as soon as debugging is done.

3. Write an absolute path in ECC_MCP_CONFIG_PATH

~ is not expanded in a launchd plist. Writing /Users/your-home-directory-name/.claude/settings.json literally is the most reliable. path.resolve is a Node feature and does not perform the shell's tilde expansion (lines 55–61). A typo makes all server configuration disappear.

4. Register aliases for cache inspection, reset, and log monitoring

So you can move immediately when a problem occurs, put aliases in .zshrc.

alias mcp-health="python3 -m json.tool ~/.claude/mcp-health-cache.json"
alias mcp-reset="echo '{\"version\":1,\"servers\":{}}' > ~/.claude/mcp-health-cache.json"
alias mcp-logs="tail -f ~/.claude/logs/hooks.log | grep MCPHealthCheck"

5. Explicitly list every API key a stdio server needs in config.env

Claude started from launchd doesn't have the login shell's environment variables. Enumerating all required keys in each server's env section in ~/.claude/settings.json makes interactive behavior and batch behavior match.

6. Tune the TTL to match session usage frequency

The default of 2 minutes (DEFAULT_TTL_MS = 2 * 60 * 1000, line 23) is a general-purpose value. In environments that call one server at high frequency, like overnight batches, extending it to 5–10 minutes greatly reduces probe count and raises throughput. During debugging, shrinking it to 30 seconds makes reproducing problems faster.

# 夜間バッチ環境
export ECC_MCP_HEALTH_TTL_MS=600000   # 10分

# デバッグ中
export ECC_MCP_HEALTH_TTL_MS=30000    # 30秒

7. Match the backoff base value to the server's characteristics

The default 30 seconds assumes a local process restart. For an external API server that takes 1–5 minutes to recover, raising it to ECC_MCP_HEALTH_BACKOFF_MS=60000 (1 minute) makes the recovery-check interval match reality. Conversely, if a local stdio server recovers in a few seconds, 10 seconds is plenty.

8. Manage important servers with PM2

When PM2's auto-restart combines with this hook's reconnect → re-probe cycle, recovery from transient failures becomes fully automated. The cycle is: the server crashes → PM2 restarts it → the next PreToolUse hook calls the reconnect command → PM2 returns success as "already started" → the re-probe confirms healthy → tool execution is allowed. Since moving to this combination, cases of "the server was dead the next morning" have dropped to zero.

9. Periodically check config-file resolution results via the source field

The source field on each entry in ~/.claude/mcp-health-cache.json shows which config file the server configuration was read from. After adding a .claude/settings.json to a project directory, use this field to confirm the intended config is being used. Most of "I fixed the home config but it doesn't work" is resolved here.

10. Build the habit of watching the hook's logs in real time

emitLogs (lines 561–565) outputs logs to stderr. Because they flow into Claude Code's hook log, you can check them with tail -f. If the word skipping appears, blocking is working. If connection restored appears, the reconnect succeeded. With the habit of reading the logs, you can confirm in seconds whether the hook is behaving as intended.

11. Centralize hook registration in the home config

You can write hooks in a project's .claude/settings.json too, but I strongly recommend writing the health-check hook only in home's ~/.claude/settings.json. Scattering it per project creates holes like "the hook doesn't take effect only in that project." To receive uniform protection across all projects, treat the home config as canonical.

12. When failureCount grows abnormally large, manually delete the entry to reset it

If markHealthy is called it automatically resets to failureCount: 0, but when the backoff is too long and the path to markHealthy is closed off, manual deletion is fastest. Registering the python3 one-liner above as an alias lets you reset a specific server's state in 10 seconds.

---

Summary

After being laid off and dropping to ¥0/month revenue, the thing I feared most in rebuilding my autonomous environment from scratch was the scenario of "the batch I started at night is completely wiped out by morning." What most reliably prevents that isn't a stronger model or better-polished prompts — it's a mechanism that detects failures and records them outside the context.

If I summarize the design of mcp-health-check.js in one line: "don't lose health information even when context is compacted." mcp-health-cache.json lives outside the session. The backoff calculation is handled by code. The model doesn't have to reason about "this server might be down." The tokens and attention saved can go to the actual task.

The reason the ¥1.2M/month environment keeps being ¥1.2M/month is that the overnight batches don't fall over before morning. And they don't fall over because this hook is quietly doing its job.

The implementation is 721 lines, but the core comes down to three points.

• Persist state outside the context — the record survives regardless of session compaction
• Branch strategy by status code — 429, 503, and ECONNREFUSED are treated as different things
• Cap the backoff — MAX_BACKOFF_MS's 10 minutes prevents total abandonment

Once you understand these three points, you'll be able to tune it and isolate problems on your own.

---

I've put the full picture of the mechanism, the breakdown of the ¥1.2M/month, and the 30-day procedure into a paid note.

📕 Claude Code自律環境で、実際どう稼ぐか ― 仕組み・実例・始め方・サポート

---

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

Comments

[Mads Hansen]

Useful separation of reachability from tool execution, but the state model should go one step further: reachable, authenticated, authorized, ready, and degraded are different facts. A bare GET returning 401/403/406 proves transport reachability; it should not populate the same “healthy” cache used to admit a privileged tool call. I’d also avoid putting 429 in a generic reconnect set—respect Retry-After and back off without restarting a healthy server. Cache entries should be partitioned by endpoint, credential generation/principal, and capability probe, then invalidated on token rotation or tool-catalog change. Otherwise one caller’s successful probe can mask another caller’s auth or scope failure.