DEV Community

Komal deep
Komal deep

Posted on

Connecting My Agent to SigNoz's MCP Server: A First-Timer's Debugging Diary

#signoz #wemakedev #observability #hackathon

I want to start with the number that actually matters: three hours. That's roughly how long I spent just getting a connection to succeed, out of a five-day hackathon window. If you're building something similar and hit the same walls, this post should save you most of that time.

What I was building, and why this angle

This was my first hackathon. My background is data science and NLP — classification, prediction models, the kind of ML work where inputs and outputs are known quantities. Agents, MCP servers, observability — none of that was in my vocabulary two weeks ago. So this post is written from that specific angle: what actually trips someone up coming from ML into agent-plus-observability tooling for the first time.

For Agents of SigNoz (WeMakeDevs x SigNoz), I built an agent that connects to a self-hosted SigNoz instance through its MCP server, answers natural-language questions grounded in live telemetry, and then classifies what it finds — latency, error-rate, resource-exhaustion, dependency-failure, or healthy — with a confidence score traced back into SigNoz itself.

Setting up SigNoz via Foundry 

bash
`curl -fsSL https://signoz.io/foundry.sh | bash`
Enter fullscreen mode Exit fullscreen mode 
yaml
`apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true`
Enter fullscreen mode Exit fullscreen mode 
bash
`foundryctl cast -f casting.yaml`
Enter fullscreen mode Exit fullscreen mode

This brought up SigNoz and its bundled MCP server as separate Docker containers — a detail that turned out to matter a lot more than I expected.

Bug #1: 401 Unauthorized

My first connection attempt to http://localhost:8000/mcp returned a flat 401. Self-hosted SigNoz's MCP server needs two specific headers, not the generic Bearer-token pattern I tried first:

python
`async with streamablehttp_client(
    MCP_URL,
    headers={
        "SIGNOZ-API-KEY": os.getenv("SIGNOZ_API_KEY"),
        "X-SigNoz-URL": "http://signoz-signoz-0:8080",
    },
) as (read_stream, write_stream, _):`
Enter fullscreen mode Exit fullscreen mode

Bug #2: 400 Bad Request — the Docker networking trap

Adding the headers got me past the** 401*, straight into a **400*. I initially set X-SigNoz-URL to http://localhost:8080, since that's where SigNoz's UI loaded from my own machine. But the MCP server runs inside its own container — and inside that container, localhost refers to the container itself, not my SigNoz instance. The server's own logs said it plainly:

`"msg":"Invalid X-SigNoz-URL header" ... "error":"host \"localhost\" is not allowed"`
Enter fullscreen mode Exit fullscreen mode

docker ps showed the actual container name — signoz-signoz-0 —, and that's what needed to go in the header instead. This is the single biggest lesson I'd hand to my past self: when a container-to-container request fails, check what hostname the containers actually use to reach each other on the Docker network. It's rarely localhost.

Bug #3: a plain variable-naming mistake

Once I fixed the headers, I hit a NameError: name 'read_stream' is not defined. I'd renamed the tuple unpacking on the async with line but hadn't updated where those names were used two lines later:

python
`) as (read_stream, write_stream, _):
    async with ClientSession(
        read_stream,
        write_stream,
    ) as session:`
Enter fullscreen mode Exit fullscreen mode

Small, but the traceback pointed into the MCP library's internals, not my own line — a good reminder that an error surfacing deep in someone else's stack trace can still be your own typo.

Bug #4: 403 — a permissions gap, not a connection problem

With the connection finally working, calling signoz_list_services returned:

`403: only viewers/editors/admins can access this resource`
Enter fullscreen mode Exit fullscreen mode

The API key itself was valid — the service account behind it just had no role assigned. The fix was in Settings → Service Accounts, assigning Editor or Admin. Worth remembering: a valid key and a valid role are two separate things.

What I actually built on top: reasoning, not just retrieval

SigNoz's MCP server gives you raw retrieval — service lists, traces, metrics, logs. It has no opinion about what any of that data means. Coming from a classification background, my instinct was to add exactly that missing layer:

python
`def classify_incident(user_question: str, telemetry_data: str) -> dict:
    classification_prompt = f"""You are classifying an observability finding.

User question: {user_question}
Telemetry data retrieved: {telemetry_data}`
Enter fullscreen mode Exit fullscreen mode

Classify this into exactly ONE category: latency, error-rate, resource-exhaustion, dependency-failure, healthy, or unknown.

Respond with ONLY valid JSON, nothing else:
`{{"category": "", "confidence": , "reasoning": ""}}"""

response = hf_client.chat_completion(
    model=MODEL_ID,
    messages=[{"role": "user", "content": classification_prompt}],
    max_tokens=150,
    temperature=0,
)
raw = response.choices[0].message.content.strip()
return json.loads(raw)`
Enter fullscreen mode Exit fullscreen mode

The part I actually care about is where the result goes — not just into the chat response, but into SigNoz's own trace:

python
`with tracer.start_as_current_span("incident_classification") as classify_span:
    classification = classify_incident(request.query, telemetry_data)
    classify_span.set_attribute("incident.category", classification["category"])
    classify_span.set_attribute("incident.confidence", classification["confidence"])`
Enter fullscreen mode Exit fullscreen mode

That means my agent's own judgment becomes something you can trace inside SigNoz, the same way you'd trace any other part of the system. SigNoz observes the target service; this makes it also observe the agent's reasoning about that service.

Screenshot: SigNoz trace view showing the incident_classification span with incident.category and incident.confidence attributes

What I'd tell someone starting this from an ML background

  • Docker networking will very likely be your biggest time sink — not the AI part. Budget for it honestly.
  • Read the server's logs the moment a request fails for an unclear reason — docker logs_ _ told me more in thirty seconds than an hour of guessing at headers did.
  • A service account's API key and its assigned role are two separate failure points — check both before assuming the key is broken.
  • Look for where your actual background transfers, instead of trying to become a full infra person overnight. For me that was "don't just retrieve data, judge it" — that's ML thinking applied to an observability tool, and it's the one piece of this that's genuinely mine.

What's next

With more time, I'd add a small retrieval layer surfacing similar past incidents, and wire a SigNoz alert to trigger the agent automatically instead of waiting on a question.

Conclusion

If you're self-hosting SigNoz's MCP server and hit a 401, 400, or 403 in that order — check your headers, your Docker hostnames, and your service account's role. That covered nearly everything that stood between a working agent and me, and I hope it saves you the three hours it cost me.

Top comments (0)