I've been using AI agents with MCP tools for a few months. At some point I noticed something uncomfortable: I had no idea what was happening inside them.
An agent calls a tool. Did it succeed? How long did it take? Did the agent call the same tool 6 times because it couldn't parse the response? I genuinely could not tell. The only output was either a final answer or a vague failure. Everything in between was invisible.
That invisibility felt like a real problem — not a theoretical one. So I decided to build something that fixes it, using OpenTelemetry and SigNoz. This is what I built, what I got wrong the first three times, and what finally worked.
The Actual Problem (With Specifics)
MCP uses a simple JSON-RPC 2.0 protocol over stdio. An agent sends {"method": "tools/call", "params": {"name": "search", "arguments": {...}}} and a server responds. That's it. No request IDs in logs, no timing, no error rate, no session concept.
Three failure modes kept appearing in my testing:
Loops. An agent called search with the same query 5 times in a row. I only noticed because the demo slowed down and I happened to be watching. There was no alert, no log entry, nothing. The agent was stuck in a self-reinforcing loop and the MCP layer had no opinion about it.
Hung calls. slow_tool takes about 20 seconds. The agent just waited. No timeout event. No trace. You'd only know something was wrong if you stared at the terminal long enough.
Silent drift. The fake MCP server I wrote for testing can add a new tool at runtime (I called it bonus_tool). Once it appeared in tools/list, the agent started trying to use it. But there was no record of when the server's tool list changed — it just... happened.
These aren't contrived edge cases. Loops happen when LLM output formats change slightly. Hung calls happen when downstream APIs have degraded performance. Drift happens whenever a server does a zero-downtime update. They're real.
The Architecture: One Command, No Code Changes
The core idea: don't ask people to add an SDK to their agent or their server. Just intercept the stdio stream.
AI Agent
│ stdin/stdout (JSON-RPC 2.0)
▼
MCP Observer (proxy) ──────▶ SigNoz (via OTLP gRPC)
│ passthrough
▼
MCP Server (unchanged)
You run your MCP server through the proxy:
# Before
node my-mcp-server.js
# After
node mcp-observer.js -- node my-mcp-server.js
The proxy spawns the server as a child process, wires up bidirectional stdio, and taps every newline-delimited JSON message. It's a transparent middleman — every message passes through unmodified, but the observer sees all of it.
For each tools/call request, I start an OTel span:
const span = tracer.startSpan(`mcp.tool.call:${toolName}`, {
attributes: {
'mcp.tool.name': toolName,
'mcp.session.id': session.id,
'mcp.tool.args_hash': sha256(JSON.stringify(args)), // never log raw args
'mcp.call.id': String(callId),
}
});
The span closes when the response arrives. Duration is captured automatically.
The thing I almost got wrong: I almost logged raw arguments as a span attribute. That's a bad idea — tool arguments often contain user queries, file paths, or credentials. SHA-256 hashing the arguments lets you correlate "this is the same call repeated 4 times" without exposing what was actually in them.
Building the Three Detectors
Loop Detector
const key = `${session.id}:${toolName}:${argsHash}`;
const calls = callHistory.get(key) ?? [];
calls.push(Date.now());
const recent = calls.filter(t => t > Date.now() - WINDOW_MS);
callHistory.set(key, recent);
if (recent.length >= THRESHOLD) {
emitAnomaly('mcp.anomaly.loop_detected', { ... });
}
The threshold I landed on: 3 identical calls within 60 seconds. Too tight and you get false positives. Too loose and you're notifying after the damage is done. 3/60 caught every real loop in testing without any false positives.
Hung-Call Detector
The trick here is that you want to fire while the call is still in flight — not after it resolves. This rules out measuring duration at span close time. Instead, I set a setTimeout when the call starts and cancel it if a response arrives before it fires:
const timer = setTimeout(() => {
emitAnomaly('mcp.anomaly.hung_call', {
'tool.name': toolName,
'elapsed_ms': THRESHOLD_MS,
});
}, THRESHOLD_MS);
// On response: clearTimeout(timer)
// Then also emit a 'hung_call_resolved' with final duration
The hung_call_resolved event turned out to be important. It tells you not just that a call was slow but whether it eventually succeeded. A call that hung for 18s and then returned successfully is very different from one that hung and then errored. Both fire the initial alert, but the resolution event differentiates them.
Drift Detector
const prev = lastToolList.get(session.id) ?? [];
const next = response.tools.map(t => t.name);
const added = next.filter(n => !prev.includes(n));
const removed = prev.filter(p => !next.includes(p));
if (added.length || removed.length) {
emitAnomaly('mcp.anomaly.capability_drift', { added, removed });
}
lastToolList.set(session.id, next);
This one is simple but took me a moment to figure out the right comparison point. You want to diff per-session, not globally — different sessions can legitimately connect to different server versions. If you diff globally, you get noise.
Adding Gemini AI Diagnosis — and the Mistake I Almost Made
Once the detectors were working, I added Gemini AI to generate plain-English root cause summaries. The first implementation I wrote awaited the Gemini call before emitting the anomaly log. That was wrong.
Gemini sometimes takes 500–800ms. The proxy is in the hot path of every tool call response. If I await the AI call, I'm adding latency to the agent for every anomaly. The agent doesn't care why it was slow — it cares about getting the response.
The fix: fire and forget.
// This runs, then immediately moves on — does NOT await
generateRootCause(anomalyType, context).then(summary => {
// Emits a separate log event with the AI diagnosis
emitAnomaly(`${anomalyType}.ai_diagnosis`, {
'root_cause_ai_summary': summary,
'diagnosis_source': 'gemini',
});
});
The anomaly log fires immediately. The AI diagnosis arrives 500ms later as a separate correlated log event. Both carry the same trace_id and span_id, so they appear together in SigNoz when you filter by trace.
Here's the actual Gemini output I got for the hung call (pulled directly from SigNoz Logs Explorer — the root_cause_ai_summary attribute):
"Session 893fffd4 experienced a hung call on slow_tool after exceeding the 15,000ms threshold, following a rapid sequence of eight successful lightweight operations. This sudden latency spike strongly indicates an unhandled upstream timeout, database lock, or deadlock. Check downstream server logs for call ID 9 to see where execution stalled, and consider implementing a circuit breaker."
That's genuinely useful. It has the session ID, call ID, and a concrete suggestion. That's what I wanted — not "error occurred", but "here's what probably happened and what to do about it."
SigNoz: What I Actually Used
Traces Explorer. Filtering service.name = mcp-observer shows every tool call span. Clicking into mcp.tool.call:slow_tool shows the 15-second duration immediately. The span attributes panel shows the session ID and argument hash.
Logs Explorer. The filter body LIKE 'mcp.anomaly.%' shows all anomaly events. Each row is clickable — the sidebar reveals the full structured attributes including root_cause_summary and root_cause_ai_summary. Clicking the trace link in the sidebar jumps to the exact span that triggered the anomaly.
One thing that caught me: the body field on log records is just the event name (e.g., mcp.anomaly.hung_call). The full diagnostic text lives in attributes_string → root_cause_ai_summary. I spent longer than I'd like to admit looking for the text in the wrong field.
Trace ↔ Log correlation. This took the most wiring. To get trace_id and span_id to appear as top-level columns on log records (which SigNoz uses for the "jump to trace" button), you need to emit the log inside the span's active context:
context.with(trace.setSpan(context.active(), span), () => {
anomalyLogger.emit({ body: eventName, attributes: {...} });
});
If you emit outside the context, the log record gets no trace link. This is not obvious from the OTel docs and I only found it by inspecting the log records in SigNoz and noticing trace_id was empty.
What I'd Tell Someone Starting This
Start with one detector. I tried to build all three at once and tangled the session state. The loop detector alone is a complete, testable unit. Get that working end-to-end into SigNoz, then add the others.
SHA-256 your arguments. Don't log raw tool arguments. You'll regret it when you realize they contain user data.
Non-blocking AI enrichment from the start. Don't architect yourself into a corner where you have to refactor later. Fire-and-forget is the right pattern for AI calls in any hot path.
Check your actual span names before building dashboards. I spent time building dashboard panels that filtered for name = 'execute_tool' — a span name I made up. My proxy actually emits mcp.tool.call:search, mcp.tool.call:slow_tool, etc. Running this in SigNoz Logs told me the truth immediately:
SELECT DISTINCT name FROM signoz_traces
WHERE serviceName = 'mcp-observer'
Always verify what you're actually emitting before you build anything on top of it.
Conclusion
MCP is new enough that its observability story is essentially blank. That's either a problem or an opportunity, depending on how you look at it. I chose opportunity.
The proxy works. Every anomaly fires, every AI diagnosis lands in SigNoz, and the trace-log correlation makes debugging a session a matter of clicking rather than grepping.
If you're building with MCP tools and you want to see what your agents are actually doing, the code is at https://github.com/Arjun-3105/mcp-observer. npm run demo gets you all three anomaly types firing in about 90 seconds.
One command. No code changes to your agent or server. Full visibility.
Top comments (0)