Everything Was Green. Nothing Was Listening. — Self-Hosting SigNoz for MCP Tool-Error Traces

Thirumalaiboobathi B — Fri, 17 Jul 2026 13:17:34 +0000

Everything was green. signoz-signoz-0 was healthy. The UI answered on localhost:8080 with a clean HTTP 200. The collector logs said, in so many words, everything is ready.

Nothing was listening on 4318.

This is the story of two days spent self-hosting SigNoz to validate a library I'd published a week earlier, and what I found instead. Spoiler: the library works. The traces are shaped correctly. I still don't have a screenshot of them in the SigNoz UI, and I'd rather tell you why than pretend otherwise.

The thing I was trying to prove

I maintain opentel-mcp, a one-line OpenTelemetry instrumentation library for Model Context Protocol servers. Its one genuinely uncommon behaviour: MCP's CallToolResult can come back with isError: true while the JSON-RPC call itself succeeds. The transport is fine. The tool failed. Most instrumentation sees a 200-equivalent and moves on.

opentel-mcp catches that case and marks the span status: ERROR with error.type: tool_error. As far as I can tell it's the only Node library doing this, though I'd be happy to be corrected.

That's the claim. A claim like that deserves a real backend behind it, not just spans printed to stderr. SigNoz is OpenTelemetry-native and self-hostable, so it seemed like the honest test: point the exporter at a real ingester, open the trace explorer, and see whether the error actually surfaces the way I say it does.

The demo

Small on purpose. One McpServer, two tools:

echo — returns cleanly. Should produce an OK span.
fetch_weather — always returns isError: true. Should produce the ERROR span with error.type: tool_error.

Instrumented in one call:

instrumentMcpServer(server, {
  setupNodeSdk: true,
  exporterUrl: 'http://localhost:4318/v1/traces'
});

One note that matters more than it looks: opentel-mcp only ships @opentelemetry/exporter-trace-otlp-http. There's no gRPC exporter. So exporterUrl has to be the OTLP/HTTP URL with the full path — http://localhost:4318/v1/traces — not the bare 4317 gRPC port. Every backend guide lists both ports side by side and it's an easy thing to get wrong at 1am.

Self-hosting SigNoz: four things nobody told me

1. `deploy/docker/docker-compose.yaml` doesn't exist anymore

Most of the guides you'll find (including the one in my own head) say clone the repo, cd deploy/docker, docker compose up -d. That path is gone. SigNoz has moved to Foundry:

curl -fsSL https://signoz.io/foundry.sh | bash
# write a casting.yaml
foundryctl cast -f casting.yaml

Not worse, just different, and different in a way that silently invalidates every blog post written before it. Which is a nice bit of irony for a post like this one.

2. My machine was under spec and Docker didn't say so

7.28GB total RAM. Docker Desktop's WSL2 VM had helped itself to 3.47GB of it. SigNoz's documented minimum is 4GB.

What that looks like in practice is not an error message. It looks like signoz-signoz-0 dying partway through a migration with unexpected EOF. I bumped the VM to 5GB via .wslconfig and wsl --shutdown, which left Windows about 2.3GB to think about its life choices. Tight, but it got the container up.

3. The crash left a lock behind, and the lock outlived the container

Here's the fun part. That first crashed migration wrote a row into migration_lock. Postgres data lives on a volume, so the lock survived every restart afterwards and blocked each new boot. The container wasn't failing for the original reason anymore — it was failing because of the corpse of the original reason.

docker exec signoz-metastore-postgres-0 \
  psql -U signoz -d signoz -c "DELETE FROM migration_lock;"
docker restart signoz-signoz-0

I had to do this twice. Restarting Docker Desktop doesn't clear it. The volume persists, so the lock persists.

4. Two self-inflicted wounds worth naming

I ran npm install from inside the example directory instead of the repo root. That gave the demo its own second copy of @modelcontextprotocol/sdk and @opentelemetry/api. Spans stopped printing. Not an error — just silence, which is the worst possible failure mode when you're debugging an observability tool. Classic dual-package hazard. Fixed by registering the example as a proper workspace member so everything shares one copy.

Then: BatchSpanProcessor's flush timer is unref()'d. For a short-lived script like this demo, the process can exit before the batched export ever fires, dropping every span silently. stderr output is unaffected because it's synchronous — so it looks fine locally while nothing reaches your backend. Fixed by flushing on stdin close.

Both of those are on me, not on SigNoz. Both are exactly the kind of thing that would have had me blaming the backend for an hour.

The part where it works

With all that cleared, the demo produces exactly what it should:

{
  name: 'tools/call echo',
  attributes: {
    'mcp.method.name': 'tools/call',
    'gen_ai.operation.name': 'execute_tool',
    'gen_ai.tool.name': 'echo',
    'mcp.tool.argument_count': 1,
    'jsonrpc.request.id': '1'
  },
  status: { code: 1 }
}
{
  name: 'tools/call fetch_weather',
  attributes: {
    'mcp.method.name': 'tools/call',
    'gen_ai.operation.name': 'execute_tool',
    'gen_ai.tool.name': 'fetch_weather',
    'mcp.tool.argument_count': 1,
    'jsonrpc.request.id': '2',
    'error.type': 'tool_error'
  },
  status: { code: 2 }
}

status: { code: 1 } is OK. status: { code: 2 } is ERROR, carrying error.type: 'tool_error'. The tool failure got caught. The claim holds.

The wall

Spans generated correctly. Exporter configured correctly. Backend healthy, UI up, collector satisfied with itself.

Export times out.

Before blaming anything I checked whether the port was actually open, from inside the network:

docker run --rm --network signoz-network curlimages/curl:latest \
  -X POST http://signoz-ingester-1:4318/v1/traces \
  -H "Content-Type: application/json" -d '{"resourceSpans":[]}'
# HTTP 000 — connection refused

And then, because container networking lies to you often enough to warrant it, straight from the process table inside the ingester:

docker exec signoz-ingester-1 sh -c "cat /proc/net/tcp"
# no listener on 4318

No listener. Not a firewall, not a DNS thing, not my exporter. The OTLP receiver simply was never bound.

The on-disk ingester.yaml is correct — the receiver is right there in the config file. But the ingester runs with --manager-config=/etc/opamp-config.yaml, which means it doesn't use that file as its source of truth. It fetches its effective config remotely via OpAMP from the signoz service. That handshake kept erroring, and on eventually connecting it appears to apply a fallback default config with no otlp receiver in it.

So the failure mode is: correct config on disk, ignored; broken config delivered over the wire; every health check green; nothing listening. Observability platform, unobservable.

My leading hypothesis is version skew between two independently-tagged :latest images — the ingester and the OpAMP server disagreeing about config schema, with the ingester falling back rather than failing loudly. I want to be clear about my confidence here: I'm certain about the symptom, and I've verified it three ways. The root cause is my best reading of the evidence, not a confirmed diagnosis. If someone from SigNoz reads this and tells me I cast a bad casting.yaml, that's a completely plausible outcome and I'd like to know.

What I shipped anyway

When the flush hit an unreachable ingester, my demo died with an uncaught exception and a stack trace out of http-transport-utils.js. Ugly, and it points the finger at the wrong place — someone hitting that would reasonably assume the library broke.

So I wrapped the shutdown path. Now it exits 0 with:

opentel-mcp: failed to flush spans to SigNoz: socket hang up

One line. Correct blame. A backend being unreachable is a normal condition, not a crash, and a library that instruments error handling should probably model decent error handling.

That fix is the one lasting artifact of two days of infrastructure debugging, and honestly it's a better outcome than the screenshot I came for.

The feature I liked most, which I never got to use

An awkward heading for a post that ends without a trace explorer screenshot. But my honest answer is the thing that caused the failure: OpAMP-managed remote configuration.

Think about what it's for. Fleets of collectors, centrally configured, no redeploy to change what you're collecting. That's a genuinely good idea and it's why the on-disk config was being ignored — the system was working exactly as designed, just with the wrong config on the other end of the wire. Static-config observability stacks don't have this failure mode, but they also don't let you retune a hundred collectors from one place.

The lesson I'm taking is narrower and more useful than "SigNoz is broken," because I don't think it is: when your config is delivered rather than mounted, "the config file is correct" stops being evidence of anything. Check the listener, not the YAML.

Repro, if you want to try

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

# if signoz-signoz-0 crash-loops:
docker exec signoz-metastore-postgres-0 \
  psql -U signoz -d signoz -c "DELETE FROM migration_lock;"
docker restart signoz-signoz-0

# from opentel-mcp repo root, NOT the example dir:
npm install
cd examples/signoz-demo && printf '...' | node server.js

And before you trust the trace explorer's emptiness to mean anything about your own code:

docker exec signoz-ingester-1 sh -c "cat /proc/net/tcp"

Full friction notes are in examples/signoz-demo/README.md.

If you've self-hosted SigNoz via Foundry recently and your ingester did bind its OTLP receiver, I'd genuinely like to hear what your casting.yaml looked like — there's a decent chance the answer is embarrassing and mine.

opentel-mcp is on npm and GitHub.

Building AgriNova AI Assistant – My First Step into GenAI + Cloud

Thirumalaiboobathi B — Thu, 28 May 2026 04:57:19 +0000

Recently, I built AgriNova AI Assistant🌾🤖 — an AI-powered agriculture assistant designed to support farmers and agriculture-related guidance using Generative AI.

This project became a major learning milestone for me because it combined:

☁️ Cloud
🤖 Generative AI
⚙️ DevOps concepts
🧠 AI agents
🌐 Full Stack integration

💡 What AgriNova AI Can Do

Provide agriculture-related guidance
Answer farming queries using AI
Generate smart responses using LLMs
Work as an AI assistant/chat system
Explore privacy-focused AI workflows

📚 What I Learned

This project helped me understand:

How AI assistants actually work
Basics of prompt engineering
Integrating AI models into applications
Running local LLMs with Ollama
Connecting cloud + AI workflows
Importance of scalable architecture

Biggest Takeaway

Earlier, I used to spend more time only consuming content and scrolling.

Now I’m trying to build consistently, learn publicly, and improve through real projects.

This is just the beginning of my AI + Cloud journey ✨

Big thanks to the AWS and developer communities that motivated me to start building and sharing publicly 🙌

DEV Community: Thirumalaiboobathi B