Catching Silent Failures in MCP Servers with OpenTelemetry and SigNoz

The Problem: MCP Tools Can Lie

The Model Context Protocol (MCP) has become the connective tissue between LLM agents and the tools they use. But MCP has a subtle observability gap that most instrumentation misses entirely.

Here's the failure mode: an MCP tool call can return isError: true inside a perfectly successful HTTP 200 / JSON-RPC response. From the transport layer's point of view, nothing went wrong. The HTTP status is 200. The JSON-RPC envelope reports success. The trace span, in most instrumentations, is marked OK.

But the tool itself failed. The agent got a bad answer back. And nobody knows.

This is the class of bug that quietly erodes trust in AI agents: silent, non-retryable, invisible in every dashboard, discovered only when a downstream user complains.

That's the problem opentel-mcp was built to solve — and building the SigNoz observability story around it for this hackathon is what turned a library into a full end-to-end demonstration.

What I Built

Three things came together during the hackathon week:

1. opentel-mcp v0.3.0 — an open-source npm library that adds OpenTelemetry instrumentation to MCP servers (Node.js / TypeScript), with silent-failure detection at its core.
2. A "MCP Tool Health" dashboard in SigNoz — five Query Builder panels that surface the failures other instrumentation misses.
3. A real ingester bug found, reported, and worked around — because you don't get real observability without a real backend.

Below is what each of those looked like in practice.

---

Part 1: The Library — opentel-mcp v0.3.0

The core detection logic is small but load-bearing. When a CallToolResult comes back, the library inspects the payload itself — not just the transport response — and if isError === true, the span is marked:

• status: ERROR
• error.type: tool_error

That single check is the differentiator. Every other Node MCP instrumentation library I benchmarked treats a 200 as a success, full stop.

v0.3.0 added metrics on top of the traces:

• mcp.tool.calls — total call counter
• mcp.tool.errors — transport-level and JSON-RPC-level errors
• mcp.tool.silent_failures — the isError-in-200 case, isolated
• mcp.tool.duration — histogram, tagged with outcome=success|error|silent_failure

Critically, both traces and metrics are driven by the same detection logic. They can never disagree about what failed — which is a class of observability bug in its own right, and one I explicitly designed out.

Attributes follow the OpenTelemetry MCP semantic conventions (gen_ai.tool.name, error.type), so nothing here is proprietary — it lands cleanly in any OTel backend.

A story from the v0.3.0 release that felt too fitting to leave out: while verifying my own README examples against a real OTLP collector before publishing, I discovered that one example was silently sending traces to a no-op tracer. A silent failure, in the documentation, of a library built to catch silent failures. Fixed, re-verified, shipped. Every example in the README is now tested copy-paste-as-written.

Part 2: Wiring It Into SigNoz — and Finding a Real Bug

The plan was simple: run SigNoz self-hosted, point opentel-mcp at the OTLP receiver on 4318, watch traces land. In practice, 4318 never bound.

The ingester logs said "Everything is ready." curl http://127.0.0.1:4318/v1/traces refused connections. Something was off.

Root cause: the ingester service in the SigNoz compose.yaml was starting the collector with --manager-config=/etc/opamp-config.yaml, putting it under OpAMP remote management pointed at ws://signoz-signoz-0:4320/v1/opamp. The OpAMP handshake was failing most of the time ("Server returned an error response"), and on the occasional successful handshake, the OpAMP-supplied default configuration was clobbering the local otlp receiver definitions. So 4317 and 4318 never came up, despite the reassuring startup logs.

Fix: dropped --manager-config and --copy-path from the ingester command so it runs off the static ingester/ingester.yaml, which already correctly defines the OTLP receivers on grpc:4317 and http:4318.

After docker compose up -d --force-recreate ingester:

• Logs showed Starting HTTP server on [::]:4318 — no restart loop
• curl -X POST http://127.0.0.1:4318/v1/traces returned 200 {"partialSuccess":{}}

I reported this to Nagesh from the SigNoz team with diagnostics — happy to see the library's whole reason for existing (surface hidden failures) apply to its own backend on day one.

Part 3: Verifying the Detection End-to-End

Before touching the dashboard, I wanted proof that the silent-failure detection actually reached storage — not just stderr, not just the trace exporter, but the row that SigNoz's UI would eventually read.

I ran the existing examples/signoz-demo/server.js demo, which exposes two tools:

• echo — always succeeds
• fetch_weather — always returns isError: true

Then I queried ClickHouse directly:

trace_id	name	status	has_error	error.type
2780…	tools/call fetch_weather	Error	true	tool_error
d4fa…	tools/call echo	Ok	false	—

Exactly the outcome the library is supposed to produce: the silent failure was caught, labeled, and stored with the semantic-convention attribute set — not the successful-looking span the transport layer would have reported on its own.

Part 4: The Dashboard — MCP Tool Health

With traces landing correctly, the dashboard came together as five panels, all built with SigNoz Query Builder (no default templates):

1. Tool call volume by tool — count() grouped by gen_ai.tool.name, filtered to mcp.method.name = 'tools/call'
2. Tool call latency (p95) by tool — p95(duration_nano) with the same grouping
3. Error rate: transport-level vs silent failures — two series in one panel, one for has_error = true AND error.type != 'tool_error', one for error.type = 'tool_error'
4. Silent failures by tool — dedicated panel, filtered to error.type = 'tool_error' only
5. Silent failure rate — a stat panel with the formula (silent_failures / total_calls) * 100

To ground the query schema in reality rather than guesswork, I cross-referenced it against the SigNoz source I had checked out — pkg/types/dashboardtypes/dashboard.go, pkg/telemetrytraces/const.go, pkg/types/telemetrytypes/field_context.go, grammar/FilterQuery.g4, and a real shipped fixture at pkg/sqlmigration/088_migrate_ii_dashboards/redis/overview.json. This version's dashboard schema was new enough that I didn't want to trust general knowledge.

I then validated the dashboard JSON against the live SigNoz API: POST /api/v1/dashboards returned a clean 401 unauthenticated — not a 400 schema error. That confirmed the JSON was routing and parsing correctly, only gated on login. The final import through the UI worked on the first try, and all five panels rendered with real data.

The result: a silent failure rate of ~43% under mixed traffic — visible, quantified, and alertable. On any other Node MCP instrumentation, the same traffic would report ~0% errors.

What This Demonstrates

Pulling the whole thing together, this project is about three ideas working in the same direction:

• A missing detection, plugged: MCP silent failures now show up as real errors in traces and metrics, with the semantic-convention attributes any OTel backend expects.
• A real observability story on SigNoz, from ingestion through storage to a Query Builder dashboard — including a real ingester bug found and reported along the way.
• Traces and metrics that can't lie to each other, because they're driven by the same detection logic — closing a subtle class of observability bug that shows up in most dual-signal setups.

What's Next

• Formalize the structural fingerprinting approach as ADR 005 in the library
• Publish a Grafana-compatible export of the dashboard
• Ship an alerting recipe for silent-failure rate thresholds in SigNoz
• Continue collaborating with the SigNoz team on the OpAMP fallback fix upstream

Built by Team Agent 4318 for the Agents of SigNoz hackathon (July 20–26, 2026).

Comments

[Mads Hansen]

Great distinction between transport success and tool failure. The next observability gap is semantic false success: isError: false with an empty, truncated, stale, or wrong-tenant result. I’d add boundary validators and an outcome taxonomy such as protocol_error, tool_error, contract_violation, business_rejection, and success, then link the tool span to the agent step and final user outcome. Keep tool names low-cardinality and arguments redacted or hashed; use trace IDs as exemplars. Otherwise the dashboard can still report healthy tools while the workflow is wrong.