DEV Community: Thirumalaiboobathi B

The Retry Loop That's Tripling Your Bedrock Bill

Thirumalaiboobathi B — Mon, 03 Aug 2026 09:34:33 +0000

Six green spans, one broken tool

A few weeks after shipping cost attribution, I was staring at a trace that looked healthy and made no sense.

Six calls to the same tool inside one session. All HTTP 200. All span status OK. Total duration about four seconds. Nothing in the trace suggested anything was wrong.

The tool had failed all six times. Same error, every call. The agent just couldn't tell.

I only found it because the cost attribution I'd built in v0.5.0 showed six InvokeModel charges for one logical operation. The traces said everything succeeded. The dollars said otherwise.

That gap is what v0.6.1 closes.

Why the agent keeps retrying

MCP has two error channels, and only one of them behaves the way you'd expect.

Protocol errors — unknown method, malformed request — come back as JSON-RPC error objects. Those propagate normally.

Tool errors don't. When a tool executes and fails, the server returns a successful JSON-RPC response with an isError flag inside the result:

HTTP/1.1 200 OK

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "isError": true,
    "content": [
      { "type": "text", "text": "connection refused: 10.0.0.5:5432" }
    ]
  }
}

Nothing throws. Nothing rejects. Generic instrumentation reads the transport status, sees 200, and marks the span OK.

Now think about what the agent sees. It gets that error text back as a normal tool result. From the model's point of view nothing failed — it just received content that reads like a complaint about its input. So it does the sensible thing: rephrases the arguments and tries again.

The tool fails identically. The agent tries again.

Each retry resends the accumulated context to Bedrock, and the context has grown by the previous failed result. So the retries get progressively more expensive:

Attempt	Input tokens	Output tokens
1	8,000	300
2	8,400	300
3	8,800	300
4	9,200	300
5	9,600	300
6	10,000	300

Illustrative token growth for one retry loop — actual figures depend entirely on your context size and the agent's retry behaviour.

Six billed InvokeModel calls producing nothing, and not a single error span to point at.

What thrash detection emits

v0.6.1 watches for the same tool failing with the same failure fingerprint repeatedly inside one session. When that crosses a threshold, it emits one event carrying the whole loop instead of leaving it scattered across N indistinguishable spans.

Span event: mcp.loop.detected

mcp.loop.length              6
mcp.loop.wasted_tokens_in    54000
mcp.loop.wasted_tokens_out   1800
mcp.loop.wasted_cost_usd     0.19
mcp.loop.duration_ms         4310
mcp.loop.first_span_id       7b2e...
mcp.loop.first_trace_id      c81a...
mcp.loop.session_id          sess-4471
mcp.failure.fingerprint      a3f8c21d94b06e77

first_span_id is the one that saves you time at 2 AM. It points back to the span where the loop started, which is where the actual root cause lives — the other five are just echoes.

Five metrics

mcp.tool.loop.detected          Counter
mcp.tool.loop.length            Histogram
mcp.tool.loop.wasted_tokens     Histogram   tokens
mcp.tool.loop.wasted_cost_usd   Histogram   USD
mcp.tool.loop.duration          Histogram   ms

Same division of labour as v0.5.0: the counters give you alerting and trend lines, the span event gives you the drill-down when an alert fires.

The fingerprint is what makes grouping work

Detection keys off the failure fingerprinting from v0.4.0, and this is the part that makes it useful rather than noisy.

Raw error strings don't group. The same broken connection produces a different message every time — different IP, different request ID, different path. So the fingerprint normalises the error through a pipeline that strips UUIDs, paths, numbers, and hex strings, then hashes the result and truncates to 16 hex characters:

connection refused: 10.0.0.5:5432   →  a3f8c21d94b06e77
connection refused: 10.0.0.7:5432   →  a3f8c21d94b06e77
timeout after 30000ms on req_88a1   →  6d10b4e7c2f3a915
timeout after 30000ms on req_91c4   →  6d10b4e7c2f3a915

Same root cause, same fingerprint, regardless of the incidental detail. Six failures with six different messages collapse into one detected loop — which is the correct reading, because it is one problem.

This is also why v0.6.1 couldn't have shipped before v0.4.0 and v0.5.0. The loop event needs the fingerprint to know the failures are the same, and the cost attribution to say what the loop cost. Neither feature could produce this alone.

Instrumenting a Bedrock-backed MCP server

No new setup. If you're already running the library, thrash detection is on by default:

import { instrumentMcpServer } from "opentel-mcp";

const server = instrumentMcpServer(mcpServer, {
  serviceName: "mcp-server"
});

Any tool that fails three times with the same fingerprint inside 60 seconds gets flagged automatically.

Here's a Bedrock tool that will thrash if the downstream table goes missing:

import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
import { instrumentMcpServer } from "opentel-mcp";

const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });

server.tool("lookup_customer", async ({ query }) => {
  try {
    const rows = await db.query(query);
    const response = await bedrock.send(new InvokeModelCommand({
      modelId: "amazon.nova-pro-v1:0",
      body: JSON.stringify({
        messages: [{ role: "user", content: [{ text: summarize(rows) }] }]
      })
    }));

    const result = JSON.parse(new TextDecoder().decode(response.body));

    return {
      content: [{ type: "text", text: result.output.message.content[0].text }],
      _meta: { usage: result.usage, model: "amazon.nova-pro-v1:0" }
    };
  } catch (err) {
    // This is the shape that goes unnoticed: HTTP 200, isError in the payload
    return {
      isError: true,
      content: [{ type: "text", text: `query failed: ${err.message}` }]
    };
  }
});

instrumentMcpServer(server, { serviceName: "customer-tools" });

Rename the underlying table and the agent will retry this four to six times before giving up, charging Bedrock for each attempt. With v0.6.1 you get one mcp.loop.detected event with the total.

Configuration

Every field is overridable in code or by environment variable, which makes it tunable per environment without a redeploy:

Option	Env var	Default	Description
`enabled`	`OTEL_MCP_THRASH_ENABLED`	`true`	Disables detection entirely
`threshold`	`OTEL_MCP_THRASH_THRESHOLD`	`3`	Same-fingerprint failures before triggering
`windowMs`	`OTEL_MCP_THRASH_WINDOW_MS`	`60000`	Rolling window failures must fall inside
`maxTrackedKeys`	`OTEL_MCP_THRASH_MAX_TRACKED_KEYS`	`1000`	LRU cap on the tracking store
`entryTtlMs`	`OTEL_MCP_THRASH_ENTRY_TTL_MS`	`900000`	How long an idle key survives
`reEmitAfter`	`OTEL_MCP_THRASH_RE_EMIT_AFTER`	`3`	Re-emit every N failures past threshold
`assumeSingleSession`	`OTEL_MCP_THRASH_ASSUME_SINGLE_SESSION`	`false`	Opt-in fallback for undetectable transports

Invalid environment values fall back to the default silently. They never throw.

Four design decisions worth explaining

High-cardinality attributes never touch metric labels

Fingerprints are unbounded — every new bug is a new fingerprint, permanently. Session IDs are worse. Put either on a metric dimension and you create a new time series per bug and per session, forever. In CloudWatch that gets expensive fast; in any backend it eventually falls over.

So every metric carries only gen_ai.tool.name. The fingerprint and session ID live on the span event, where high cardinality is safe and you get the full detail anyway when you drill in. The allowlist is enforced in code, not by convention, because "remember not to add attributes here" is not a strategy that survives contact with a future version of yourself.

The tracking store is bounded, and there's no timer

Loop detection has to remember recent failures, which means it holds state. MCP servers on stdio transport run for the lifetime of the process — sometimes weeks. An unbounded map here would be a slow memory leak that only shows up in the longest-running deployments, which are exactly the ones you least want to debug.

So it's an LRU with a TTL: capped keys, lazy expiry on read, and an amortised sweep on write. Deliberately no setInterval — a live timer keeps the Node event loop alive and stops the server exiting cleanly, which is its own subtle bug.

Sessions are never merged on a guess

This is the setting most likely to get misconfigured, so it's worth being explicit.

Loop detection needs a session boundary. Merge two clients' failures into one bucket and you get a phantom loop: three unrelated clients each failing once looks identical to one client failing three times.

Session-oriented transports provide a real session ID. Stdio doesn't — there's exactly one connection for the process lifetime instead. The resolution order: a real session ID always wins and permanently marks the server as session-aware. Once a server has been observed handing one out, a later call without one is skipped rather than merged. A generated fallback is used only when the transport is structurally confirmed single-connection, or when you explicitly opt in via assumeSingleSession.

Otherwise detection is skipped silently rather than guessing. Skipping produces missing data; guessing produces wrong data. Wrong is worse.

It observes. It does not intervene.

Same principle as the budget flags in v0.5.0. Detecting a loop sets attributes and emits metrics. It does not cancel the request, break the connection, or refuse the next call.

An instrumentation library that can interrupt agent execution is a library that can take down production in a way nobody predicted. Loop-breaking belongs in the agent framework or an AI gateway, where it's a deliberate part of the request path and can be disabled without redeploying your MCP server.

Checking it without a collector

If you just want to know whether anything is thrashing right now, there's an in-process accessor that involves no OpenTelemetry at all:

console.log(server.getThrashSummary());

// {
//   activeLoops: 1,
//   totalLoopsDetected: 4,
//   totalWastedCostUsd: 0.09,
//   totalWastedTokensIn: 3600,
//   totalWastedTokensOut: 900,
//   topOffenders: [
//     { toolName: 'lookup_customer', fingerprint: 'a3f4c8e2b1d09f77',
//       loops: 3, wastedCostUsd: 0.03 }
//   ]
// }

Nothing is sent anywhere. It's safe to call from a health check handler, and it works before you've stood up any backend.

One caveat worth stating: activeLoops and topOffenders reflect only what's currently in the bounded store, so an evicted or expired loop won't appear even though it really happened. The cumulative totals survive both and answer "how much has this process wasted since it started." Read the totals for accounting, not the offender list.

Common questions

Does this add latency? Detection is a hash-map lookup and a counter increment on the failure path only. Successful calls do a single clear operation. No network calls, no async work.

What if the same tool legitimately fails for different reasons? Different reasons produce different fingerprints, so they don't group and no loop fires. That's the intended behaviour — a tool failing three distinct ways is a different problem from a tool failing the same way three times.

Does it work without fingerprinting enabled? No. Detection keys off the fingerprint, so with fingerprinting: false it silently never fires. Both are on by default.

What about schema validation errors? Those travel the JSON-RPC error channel rather than arriving as isError: true, so they're a separate detection path and aren't covered here. Worth knowing, since "the agent is calling the tool wrong" is arguably the failure mode most likely to loop. It's on the list.

Is the loop event emitted once or repeatedly? Once at the threshold, then again every reEmitAfter failures past it — so a 12-call loop emits at 3, 6, 9, and 12 rather than nine times. The wasted-cost figures are delta-accounted so re-emission doesn't double-count.

The gap I haven't closed

Worth stating plainly, because it's the most interesting unsolved problem in the library right now.

If the observation path itself is broken — no provider registered, collector unreachable, exporter silently dropping — then a tool failure and a clean run produce identical output. Zero. "Nothing failed" and "nothing was observed" collapse into the same state.

This came out of a review from an external reader on the v0.3.0 release post, and he was right. A liveness signal can't travel over the channel whose liveness is in question, so this has to surface on the in-process path rather than through spans.

The proposed contract is three states rather than two: OBSERVED_CLEAN, OBSERVED_FAILING, OBSERVATION_UNAVAILABLE. It's currently in the repo as a skipped specification test with the reasoning documented, rather than quietly ignored. Detecting an unbound observation path without coupling to unstable SDK internals is the part I don't have an answer for yet.

Anyone who has chased a collector that looked perfectly healthy while binding no ports will recognise the shape.

What's next

Splitting protocol-level failures from execution failures in the fingerprint, so a schema error and an upstream outage read differently
Tool schema drift detection — hashing tools/list schemas and flagging silent changes
The observation liveness contract above
Cost-aware sampling, so expensive traces survive sampling decisions that cheap ones don't

Try it

npm install opentel-mcp

npm: https://www.npmjs.com/package/opentel-mcp
GitHub: https://github.com/Thirumalaiboobathi/opentel-mcp
Docs: https://opentel-mcp-site.pages.dev/

If you're running MCP servers on AWS, the thing worth checking today is whether your tool failures are actually reaching your traces. Mine weren't — and I'd like to know what your retry patterns look like, because the edge cases people report are what shape the next release.

Track Bedrock Token Cost Per MCP Tool Call in OpenTelemetry

Thirumalaiboobathi B — Wed, 29 Jul 2026 10:41:33 +0000

Your MCP traces already tell you what was slow and what broke. Here's how to make them tell you what it cost — before the Amazon Bedrock bill shows up.

The bill that arrives three weeks late

Last month I was load-testing an MCP server that fronts a handful of Amazon Bedrock calls. Latency looked fine. Error rate looked fine. Every span was green.

Then the Cost Explorer chart came in and one tool was responsible for most of the model spend.

I went back to the traces to figure out which one. The traces had no idea. They had duration, status code, tool name, and nothing about tokens or dollars. The information I needed had already been thrown away three weeks earlier.

That gap is what this post is about, and what I shipped in opentel-mcp v0.5.0 to close it.

Why standard MCP instrumentation misses cost entirely

A typical MCP server sits in the middle of a chain like this:

User → MCP Client → MCP Server → Tool → Amazon Bedrock / Anthropic / OpenAI / Gemini

Standard OpenTelemetry instrumentation wraps the MCP server layer and records:

request latency
error status
success rate

All useful. But the model call happens inside the tool, one layer down, and its usage payload gets consumed and discarded when the tool builds its response. By the time the span closes, the token counts are gone.

So the questions you actually want answered at 2 AM stay unanswered:

Which tool is burning the most tokens?
Which workflow costs the most per invocation?
Which model is driving the spend?
Which user session blew past its budget?

Those answers exist only in the provider's billing dashboard, aggregated by day, with no trace ID attached. You can see that you spent money. You can't see which request spent it.

What cost attribution in a span looks like

In v0.5.0, every MCP tool span carries cost and token data as first-class attributes alongside latency and errors.

Token usage

mcp.tool.tokens.input mcp.tool.tokens.output mcp.tool.tokens.total
Cost

`mcp.tool.cost.usd

Model attribution

mcp.tool.model
gen_ai.response.model
`
That second one matters more than it looks. gen_ai.response.model is part of the OpenTelemetry GenAI semantic conventions, so if you already have a GenAI dashboard built in Grafana, SigNoz, or Amazon Managed Grafana, MCP tool spans show up in it without you touching a single panel. Emitting both is a small duplication that saves every downstream user a dashboard rebuild.

Budget signals

mcp.tool.cost.budget_exceeded mcp.tool.cost.budget_scope

Two new metrics counters
mcp.tool.tokens.total mcp.tool.cost.total

The counters give you the aggregate view for alerting and trend lines. The span attributes give you the drill-down when an alert fires. You need both — the counter tells you spend tripled, the span tells you which tool did it.

Instrumenting a Bedrock-backed MCP server

Setup is one line. If you already have the OpenTelemetry Node SDK configured, this is the entire change:

javascript
`import { instrumentMcpServer } from "opentel-mcp";

instrumentMcpServer(server);`

From that point on, every tool call emits trace context, token usage, estimated cost, model attribution, and budget signals.

Here's a Bedrock tool being instrumented end to end:

javascript
`import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";
import { instrumentMcpServer } from "opentel-mcp";

const bedrock = new BedrockRuntimeClient({ region: "us-east-1" });

server.tool("summarize_report", async ({ text }) => {
const response = await bedrock.send(new InvokeModelCommand({
modelId: "amazon.nova-pro-v1:0",
body: JSON.stringify({
messages: [{ role: "user", content: [{ text }] }]
})
}));

const result = JSON.parse(new TextDecoder().decode(response.body));

return {
content: [{ type: "text", text: result.output.message.content[0].text }],
// usage flows through to the span
_meta: { usage: result.usage, model: "amazon.nova-pro-v1:0" }
};
});

instrumentMcpServer(server);
`
The resulting span:

Span: mcp.tool.summarize_report
├─ duration: 1,842ms
├─ mcp.tool.model: amazon.nova-pro-v1:0
├─ mcp.tool.tokens.input: 4,210
├─ mcp.tool.tokens.output: 388
├─ mcp.tool.tokens.total: 4,598
└─ mcp.tool.cost.usd: 0.00363

One tool call. One line in the trace. Latency, model, tokens, and dollars in the same place — which means you can now sort your traces by cost the same way you sort them by duration.

Which models are priced out of the box

v0.5.0 ships pricing for 19 models across five providers:

Provider Coverage
Amazon Bedrock Nova family
Anthropic Claude family
OpenAI GPT family
Google Gemini family
DeepSeek DeepSeek models

If you're on a supported model, there's no pricing configuration to write.

If you're running a fine-tuned model, a self-hosted model, or you have negotiated enterprise rates, extend the defaults:

javascript
`import { instrumentMcpServer, DEFAULT_PRICING } from "opentel-mcp";

instrumentMcpServer(server, {
pricing: {
...DEFAULT_PRICING,
"internal-model": {
input: 0.002, // USD per 1K input tokens
output: 0.008 // USD per 1K output tokens
}
},
budgets: {
toolUsd: 0.10,
sessionUsd: 1.00
}
});`

Every provider returns usage in a different shape, so the default usage extractor is fully replaceable if your tool results don't match the common formats.

Four design decisions worth explaining

It observes. It does not enforce.

When budgets.toolUsd is exceeded, the library sets mcp.tool.cost.budget_exceeded on the span and lets the request through. It does not throw, block, or retry.

This one gets pushback, so here's the reasoning: an instrumentation library that can reject requests is a library that can take down production. Enforcement belongs in an AI gateway or proxy that sits in the request path deliberately and can be turned off without redeploying your server. Instrumentation should be safe to leave on forever.

It never throws.

Unknown model, malformed usage payload, missing session ID, garbage in the response — none of these produce an exception. The library records whatever it can resolve and moves on. Observability tooling that becomes a source of incidents defeats its own purpose.

Budget tracking is in-memory.

Session budgets are tracked per process and are not distributed across instances. Behind a load balancer, each instance tracks its own view.

That's a real limitation and I'd rather state it plainly than have you discover it in production. The alternative was a Redis dependency, and keeping this library dependency-free was worth more to me than distributed budget accuracy. If you need cluster-wide enforcement, aggregate the mcp.tool.cost.total counter in your backend instead — that's what the counter is there for.

Extraction is pluggable.

Rather than trying to internally support every provider's response schema forever, the usage extractor is a public API you can swap out entirely.

Why this belongs in traces and not a billing dashboard

AI FinOps is becoming its own discipline, and most of the tooling around it operates on daily aggregates — spend by account, spend by service, spend by tag.

Aggregates are fine for finance. They're close to useless for engineering. When your model spend jumps, "which day" doesn't help you. "Which tool, called by which workflow, on which model, in which session" is the thing that lets you actually fix something.

Putting cost in the same span as latency and errors means you stop treating them as separate investigations. A slow tool and an expensive tool are often the same tool, and you can now see that in one query.

Common questions

Does this add latency to my tool calls? Cost calculation is a table lookup and arithmetic on data already present in the response. There's no network call and no external service.

What if my model isn't in the pricing table? The span still records tokens and model name. Cost is omitted rather than guessed, and no exception is raised.

Does it work with existing OpenTelemetry GenAI dashboards? Yes. The library emits gen_ai.response.model alongside mcp.tool.model specifically so existing dashboards pick up MCP spans with no changes.

Can I use it with an MCP server not backed by Bedrock? Yes. Anthropic, OpenAI, Gemini, and DeepSeek pricing ship by default, and custom pricing covers anything else.

Does it block requests when a budget is exceeded? No. It records mcp.tool.cost.budget_exceeded on the span and continues.

What's next

Directions I'm considering for future releases:

cost-aware sampling, so expensive traces survive sampling decisions that cheap ones don't
cross-server trace correlation for multi-hop MCP topologies
predictive budget alerts via OpenTelemetry Events
richer AI FinOps dashboard templates

The goal isn't to become an AI gateway. It's to make sure that when you open a trace, the cost information is already sitting there.

Try it
bash
npm install opentel-mcp

GitHub: https://github.com/Thirumalaiboobathi/opentel-mcp — MIT licensed, issues and PRs welcome. A star helps more people find it.

npm: https://www.npmjs.com/package/opentel-mcp

CHANGELOG: https://github.com/Thirumalaiboobathi/opentel-mcp/blob/main/CHANGELOG.md

Every AI provider returns usage in its own shape, and the edge cases people report are what shape the next release. If your tool results don't parse cleanly, drop a comment with the payload shape or open an issue — that's the fastest way to get your provider supported.

I build open-source observability tooling for the MCP ecosystem. Next in this series: why CallToolResult.isError silently disappears from your traces, and what I found when I went looking for it on the Python side.

Catching Silent Failures in MCP Servers with OpenTelemetry and SigNoz

Thirumalaiboobathi B — Fri, 24 Jul 2026 08:16:48 +0000

The Problem: MCP Tools Can Lie

The Model Context Protocol (MCP) has become the connective tissue between LLM agents and the tools they use. But MCP has a subtle observability gap that most instrumentation misses entirely.

Here's the failure mode: an MCP tool call can return isError: true inside a perfectly successful HTTP 200 / JSON-RPC response. From the transport layer's point of view, nothing went wrong. The HTTP status is 200. The JSON-RPC envelope reports success. The trace span, in most instrumentations, is marked OK.

But the tool itself failed. The agent got a bad answer back. And nobody knows.

This is the class of bug that quietly erodes trust in AI agents: silent, non-retryable, invisible in every dashboard, discovered only when a downstream user complains.

That's the problem opentel-mcp was built to solve — and building the SigNoz observability story around it for this hackathon is what turned a library into a full end-to-end demonstration.

What I Built

Three things came together during the hackathon week:

opentel-mcp v0.3.0 — an open-source npm library that adds OpenTelemetry instrumentation to MCP servers (Node.js / TypeScript), with silent-failure detection at its core.
A "MCP Tool Health" dashboard in SigNoz — five Query Builder panels that surface the failures other instrumentation misses.
A real ingester bug found, reported, and worked around — because you don't get real observability without a real backend.

Below is what each of those looked like in practice.

Part 1: The Library — `opentel-mcp` v0.3.0

The core detection logic is small but load-bearing. When a CallToolResult comes back, the library inspects the payload itself — not just the transport response — and if isError === true, the span is marked:

status: ERROR
error.type: tool_error

That single check is the differentiator. Every other Node MCP instrumentation library I benchmarked treats a 200 as a success, full stop.

v0.3.0 added metrics on top of the traces:

mcp.tool.calls — total call counter
mcp.tool.errors — transport-level and JSON-RPC-level errors
mcp.tool.silent_failures — the isError-in-200 case, isolated
mcp.tool.duration — histogram, tagged with outcome=success|error|silent_failure

Critically, both traces and metrics are driven by the same detection logic. They can never disagree about what failed — which is a class of observability bug in its own right, and one I explicitly designed out.

Attributes follow the OpenTelemetry MCP semantic conventions (gen_ai.tool.name, error.type), so nothing here is proprietary — it lands cleanly in any OTel backend.

A story from the v0.3.0 release that felt too fitting to leave out: while verifying my own README examples against a real OTLP collector before publishing, I discovered that one example was silently sending traces to a no-op tracer. A silent failure, in the documentation, of a library built to catch silent failures. Fixed, re-verified, shipped. Every example in the README is now tested copy-paste-as-written.

Part 2: Wiring It Into SigNoz — and Finding a Real Bug

The plan was simple: run SigNoz self-hosted, point opentel-mcp at the OTLP receiver on 4318, watch traces land. In practice, 4318 never bound.

The ingester logs said "Everything is ready." curl http://127.0.0.1:4318/v1/traces refused connections. Something was off.

Root cause: the ingester service in the SigNoz compose.yaml was starting the collector with --manager-config=/etc/opamp-config.yaml, putting it under OpAMP remote management pointed at ws://signoz-signoz-0:4320/v1/opamp. The OpAMP handshake was failing most of the time ("Server returned an error response"), and on the occasional successful handshake, the OpAMP-supplied default configuration was clobbering the local otlp receiver definitions. So 4317 and 4318 never came up, despite the reassuring startup logs.

Fix: dropped --manager-config and --copy-path from the ingester command so it runs off the static ingester/ingester.yaml, which already correctly defines the OTLP receivers on grpc:4317 and http:4318.

After docker compose up -d --force-recreate ingester:

Logs showed Starting HTTP server on [::]:4318 — no restart loop
curl -X POST http://127.0.0.1:4318/v1/traces returned 200 {"partialSuccess":{}}

I reported this to Nagesh from the SigNoz team with diagnostics — happy to see the library's whole reason for existing (surface hidden failures) apply to its own backend on day one.

Part 3: Verifying the Detection End-to-End

Before touching the dashboard, I wanted proof that the silent-failure detection actually reached storage — not just stderr, not just the trace exporter, but the row that SigNoz's UI would eventually read.

I ran the existing examples/signoz-demo/server.js demo, which exposes two tools:

echo — always succeeds
fetch_weather — always returns isError: true

Then I queried ClickHouse directly:

trace_id	name	status	has_error	error.type
2780…	`tools/call fetch_weather`	Error	true	tool_error
d4fa…	`tools/call echo`	Ok	false	—

Exactly the outcome the library is supposed to produce: the silent failure was caught, labeled, and stored with the semantic-convention attribute set — not the successful-looking span the transport layer would have reported on its own.

Part 4: The Dashboard — MCP Tool Health

With traces landing correctly, the dashboard came together as five panels, all built with SigNoz Query Builder (no default templates):

Tool call volume by tool — count() grouped by gen_ai.tool.name, filtered to mcp.method.name = 'tools/call'
Tool call latency (p95) by tool — p95(duration_nano) with the same grouping
Error rate: transport-level vs silent failures — two series in one panel, one for has_error = true AND error.type != 'tool_error', one for error.type = 'tool_error'
Silent failures by tool — dedicated panel, filtered to error.type = 'tool_error' only
Silent failure rate — a stat panel with the formula (silent_failures / total_calls) * 100

To ground the query schema in reality rather than guesswork, I cross-referenced it against the SigNoz source I had checked out — pkg/types/dashboardtypes/dashboard.go, pkg/telemetrytraces/const.go, pkg/types/telemetrytypes/field_context.go, grammar/FilterQuery.g4, and a real shipped fixture at pkg/sqlmigration/088_migrate_ii_dashboards/redis/overview.json. This version's dashboard schema was new enough that I didn't want to trust general knowledge.

I then validated the dashboard JSON against the live SigNoz API: POST /api/v1/dashboards returned a clean 401 unauthenticated — not a 400 schema error. That confirmed the JSON was routing and parsing correctly, only gated on login. The final import through the UI worked on the first try, and all five panels rendered with real data.

The result: a silent failure rate of ~43% under mixed traffic — visible, quantified, and alertable. On any other Node MCP instrumentation, the same traffic would report ~0% errors.

What This Demonstrates

Pulling the whole thing together, this project is about three ideas working in the same direction:

A missing detection, plugged: MCP silent failures now show up as real errors in traces and metrics, with the semantic-convention attributes any OTel backend expects.
A real observability story on SigNoz, from ingestion through storage to a Query Builder dashboard — including a real ingester bug found and reported along the way.
Traces and metrics that can't lie to each other, because they're driven by the same detection logic — closing a subtle class of observability bug that shows up in most dual-signal setups.

What's Next

Formalize the structural fingerprinting approach as ADR 005 in the library
Publish a Grafana-compatible export of the dashboard
Ship an alerting recipe for silent-failure rate thresholds in SigNoz
Continue collaborating with the SigNoz team on the OpAMP fallback fix upstream

Built by Team Agent 4318 for the Agents of SigNoz hackathon (July 20–26, 2026).

Everything Was Green. Nothing Was Listening. — Self-Hosting SigNoz for MCP Tool-Error Traces

Thirumalaiboobathi B — Fri, 17 Jul 2026 13:17:34 +0000

Everything was green. signoz-signoz-0 was healthy. The UI answered on localhost:8080 with a clean HTTP 200. The collector logs said, in so many words, everything is ready.

Nothing was listening on 4318.

This is the story of two days spent self-hosting SigNoz to validate a library I'd published a week earlier, and what I found instead. Spoiler: the library works. The traces are shaped correctly. I still don't have a screenshot of them in the SigNoz UI, and I'd rather tell you why than pretend otherwise.

The thing I was trying to prove

I maintain opentel-mcp, a one-line OpenTelemetry instrumentation library for Model Context Protocol servers. Its one genuinely uncommon behaviour: MCP's CallToolResult can come back with isError: true while the JSON-RPC call itself succeeds. The transport is fine. The tool failed. Most instrumentation sees a 200-equivalent and moves on.

opentel-mcp catches that case and marks the span status: ERROR with error.type: tool_error. As far as I can tell it's the only Node library doing this, though I'd be happy to be corrected.

That's the claim. A claim like that deserves a real backend behind it, not just spans printed to stderr. SigNoz is OpenTelemetry-native and self-hostable, so it seemed like the honest test: point the exporter at a real ingester, open the trace explorer, and see whether the error actually surfaces the way I say it does.

The demo

Small on purpose. One McpServer, two tools:

echo — returns cleanly. Should produce an OK span.
fetch_weather — always returns isError: true. Should produce the ERROR span with error.type: tool_error.

Instrumented in one call:

instrumentMcpServer(server, {
  setupNodeSdk: true,
  exporterUrl: 'http://localhost:4318/v1/traces'
});

One note that matters more than it looks: opentel-mcp only ships @opentelemetry/exporter-trace-otlp-http. There's no gRPC exporter. So exporterUrl has to be the OTLP/HTTP URL with the full path — http://localhost:4318/v1/traces — not the bare 4317 gRPC port. Every backend guide lists both ports side by side and it's an easy thing to get wrong at 1am.

Self-hosting SigNoz: four things nobody told me

1. `deploy/docker/docker-compose.yaml` doesn't exist anymore

Most of the guides you'll find (including the one in my own head) say clone the repo, cd deploy/docker, docker compose up -d. That path is gone. SigNoz has moved to Foundry:

curl -fsSL https://signoz.io/foundry.sh | bash
# write a casting.yaml
foundryctl cast -f casting.yaml

Not worse, just different, and different in a way that silently invalidates every blog post written before it. Which is a nice bit of irony for a post like this one.

2. My machine was under spec and Docker didn't say so

7.28GB total RAM. Docker Desktop's WSL2 VM had helped itself to 3.47GB of it. SigNoz's documented minimum is 4GB.

What that looks like in practice is not an error message. It looks like signoz-signoz-0 dying partway through a migration with unexpected EOF. I bumped the VM to 5GB via .wslconfig and wsl --shutdown, which left Windows about 2.3GB to think about its life choices. Tight, but it got the container up.

3. The crash left a lock behind, and the lock outlived the container

Here's the fun part. That first crashed migration wrote a row into migration_lock. Postgres data lives on a volume, so the lock survived every restart afterwards and blocked each new boot. The container wasn't failing for the original reason anymore — it was failing because of the corpse of the original reason.

docker exec signoz-metastore-postgres-0 \
  psql -U signoz -d signoz -c "DELETE FROM migration_lock;"
docker restart signoz-signoz-0

I had to do this twice. Restarting Docker Desktop doesn't clear it. The volume persists, so the lock persists.

4. Two self-inflicted wounds worth naming

I ran npm install from inside the example directory instead of the repo root. That gave the demo its own second copy of @modelcontextprotocol/sdk and @opentelemetry/api. Spans stopped printing. Not an error — just silence, which is the worst possible failure mode when you're debugging an observability tool. Classic dual-package hazard. Fixed by registering the example as a proper workspace member so everything shares one copy.

Then: BatchSpanProcessor's flush timer is unref()'d. For a short-lived script like this demo, the process can exit before the batched export ever fires, dropping every span silently. stderr output is unaffected because it's synchronous — so it looks fine locally while nothing reaches your backend. Fixed by flushing on stdin close.

Both of those are on me, not on SigNoz. Both are exactly the kind of thing that would have had me blaming the backend for an hour.

The part where it works

With all that cleared, the demo produces exactly what it should:

{
  name: 'tools/call echo',
  attributes: {
    'mcp.method.name': 'tools/call',
    'gen_ai.operation.name': 'execute_tool',
    'gen_ai.tool.name': 'echo',
    'mcp.tool.argument_count': 1,
    'jsonrpc.request.id': '1'
  },
  status: { code: 1 }
}
{
  name: 'tools/call fetch_weather',
  attributes: {
    'mcp.method.name': 'tools/call',
    'gen_ai.operation.name': 'execute_tool',
    'gen_ai.tool.name': 'fetch_weather',
    'mcp.tool.argument_count': 1,
    'jsonrpc.request.id': '2',
    'error.type': 'tool_error'
  },
  status: { code: 2 }
}

status: { code: 1 } is OK. status: { code: 2 } is ERROR, carrying error.type: 'tool_error'. The tool failure got caught. The claim holds.

The wall

Spans generated correctly. Exporter configured correctly. Backend healthy, UI up, collector satisfied with itself.

Export times out.

Before blaming anything I checked whether the port was actually open, from inside the network:

docker run --rm --network signoz-network curlimages/curl:latest \
  -X POST http://signoz-ingester-1:4318/v1/traces \
  -H "Content-Type: application/json" -d '{"resourceSpans":[]}'
# HTTP 000 — connection refused

And then, because container networking lies to you often enough to warrant it, straight from the process table inside the ingester:

docker exec signoz-ingester-1 sh -c "cat /proc/net/tcp"
# no listener on 4318

No listener. Not a firewall, not a DNS thing, not my exporter. The OTLP receiver simply was never bound.

The on-disk ingester.yaml is correct — the receiver is right there in the config file. But the ingester runs with --manager-config=/etc/opamp-config.yaml, which means it doesn't use that file as its source of truth. It fetches its effective config remotely via OpAMP from the signoz service. That handshake kept erroring, and on eventually connecting it appears to apply a fallback default config with no otlp receiver in it.

So the failure mode is: correct config on disk, ignored; broken config delivered over the wire; every health check green; nothing listening. Observability platform, unobservable.

My leading hypothesis is version skew between two independently-tagged :latest images — the ingester and the OpAMP server disagreeing about config schema, with the ingester falling back rather than failing loudly. I want to be clear about my confidence here: I'm certain about the symptom, and I've verified it three ways. The root cause is my best reading of the evidence, not a confirmed diagnosis. If someone from SigNoz reads this and tells me I cast a bad casting.yaml, that's a completely plausible outcome and I'd like to know.

What I shipped anyway

When the flush hit an unreachable ingester, my demo died with an uncaught exception and a stack trace out of http-transport-utils.js. Ugly, and it points the finger at the wrong place — someone hitting that would reasonably assume the library broke.

So I wrapped the shutdown path. Now it exits 0 with:

opentel-mcp: failed to flush spans to SigNoz: socket hang up

One line. Correct blame. A backend being unreachable is a normal condition, not a crash, and a library that instruments error handling should probably model decent error handling.

That fix is the one lasting artifact of two days of infrastructure debugging, and honestly it's a better outcome than the screenshot I came for.

The feature I liked most, which I never got to use

An awkward heading for a post that ends without a trace explorer screenshot. But my honest answer is the thing that caused the failure: OpAMP-managed remote configuration.

Think about what it's for. Fleets of collectors, centrally configured, no redeploy to change what you're collecting. That's a genuinely good idea and it's why the on-disk config was being ignored — the system was working exactly as designed, just with the wrong config on the other end of the wire. Static-config observability stacks don't have this failure mode, but they also don't let you retune a hundred collectors from one place.

The lesson I'm taking is narrower and more useful than "SigNoz is broken," because I don't think it is: when your config is delivered rather than mounted, "the config file is correct" stops being evidence of anything. Check the listener, not the YAML.

Repro, if you want to try

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

# if signoz-signoz-0 crash-loops:
docker exec signoz-metastore-postgres-0 \
  psql -U signoz -d signoz -c "DELETE FROM migration_lock;"
docker restart signoz-signoz-0

# from opentel-mcp repo root, NOT the example dir:
npm install
cd examples/signoz-demo && printf '...' | node server.js

And before you trust the trace explorer's emptiness to mean anything about your own code:

docker exec signoz-ingester-1 sh -c "cat /proc/net/tcp"

Full friction notes are in examples/signoz-demo/README.md.

If you've self-hosted SigNoz via Foundry recently and your ingester did bind its OTLP receiver, I'd genuinely like to hear what your casting.yaml looked like — there's a decent chance the answer is embarrassing and mine.

opentel-mcp is on npm and GitHub.

Building AgriNova AI Assistant – My First Step into GenAI + Cloud

Thirumalaiboobathi B — Thu, 28 May 2026 04:57:19 +0000

Recently, I built AgriNova AI Assistant🌾🤖 — an AI-powered agriculture assistant designed to support farmers and agriculture-related guidance using Generative AI.

This project became a major learning milestone for me because it combined:

☁️ Cloud
🤖 Generative AI
⚙️ DevOps concepts
🧠 AI agents
🌐 Full Stack integration

💡 What AgriNova AI Can Do

Provide agriculture-related guidance
Answer farming queries using AI
Generate smart responses using LLMs
Work as an AI assistant/chat system
Explore privacy-focused AI workflows

📚 What I Learned

This project helped me understand:

How AI assistants actually work
Basics of prompt engineering
Integrating AI models into applications
Running local LLMs with Ollama
Connecting cloud + AI workflows
Importance of scalable architecture

Biggest Takeaway

Earlier, I used to spend more time only consuming content and scrolling.

Now I’m trying to build consistently, learn publicly, and improve through real projects.

This is just the beginning of my AI + Cloud journey ✨

Big thanks to the AWS and developer communities that motivated me to start building and sharing publicly 🙌

DEV Community: Thirumalaiboobathi B

The Retry Loop That's Tripling Your Bedrock Bill

Six green spans, one broken tool

Why the agent keeps retrying

What thrash detection emits

The fingerprint is what makes grouping work

Instrumenting a Bedrock-backed MCP server

Configuration

Four design decisions worth explaining

High-cardinality attributes never touch metric labels

The tracking store is bounded, and there's no timer

Sessions are never merged on a guess

It observes. It does not intervene.

Checking it without a collector

Common questions

The gap I haven't closed

What's next

Try it

Track Bedrock Token Cost Per MCP Tool Call in OpenTelemetry

Catching Silent Failures in MCP Servers with OpenTelemetry and SigNoz

The Problem: MCP Tools Can Lie

What I Built

Part 1: The Library — opentel-mcp v0.3.0

Part 2: Wiring It Into SigNoz — and Finding a Real Bug

Part 3: Verifying the Detection End-to-End

Part 4: The Dashboard — MCP Tool Health

What This Demonstrates

What's Next

Everything Was Green. Nothing Was Listening. — Self-Hosting SigNoz for MCP Tool-Error Traces

The thing I was trying to prove

The demo

Self-hosting SigNoz: four things nobody told me

1. deploy/docker/docker-compose.yaml doesn't exist anymore

2. My machine was under spec and Docker didn't say so

3. The crash left a lock behind, and the lock outlived the container

4. Two self-inflicted wounds worth naming

The part where it works

The wall

What I shipped anyway

The feature I liked most, which I never got to use

Repro, if you want to try

Building AgriNova AI Assistant – My First Step into GenAI + Cloud

Part 1: The Library — `opentel-mcp` v0.3.0

1. `deploy/docker/docker-compose.yaml` doesn't exist anymore