DEV Community

subanu-dev
subanu-dev

Posted on

# Building an SRE Sidekick: an AI agent that reads, acts on, and observes SigNoz

#signoz #ai #opentelemetry #observability

My submission for the Agents of SigNoz hackathon (Track 1 — AI & Agent Observability).

The problem: we bolt AI agents onto everything, then fly blind

AI agents now chain LLM calls and call tools autonomously. When one gets slow, expensive, or
wrong in production, you're stuck — you can't debug what you can't see.
Meanwhile the on-call engineer already drowns in dashboards. Adding an opaque AI agent on
top usually makes observability worse, not better.

So for the Agents of SigNoz hackathon I asked: what if the AI agent were the opposite
of a black box? What if it read your observability data to help you, took actions to
fix your monitoring, and was itself fully observable in the same platform?

That's SRE Sidekick — an AI agent that turns "hunt through ten dashboards" into "ask one
question," and that you can watch working, trace by trace, inside SigNoz.

What it does

  • Investigate — ask "which service is slow right now?" in plain English; it queries real traces/metrics/logs through the SigNoz MCP server and answers with specifics.
  • Act (with approval) — it can import an APM dashboard or create a threshold alert, gated by a human-in-the-loop Approve / Reject step (the agent pauses and waits for your click) before any change.
  • Observe itself — every agent run, LLM call, and tool call is an OpenTelemetry span; token usage and estimated cost follow the GenAI semantic conventions. The agent shows up in SigNoz as a service called sre-sidekick, right next to the app it's debugging.
  • Close the loop — when a SigNoz alert fires, a webhook triggers the agent to auto-investigate and post a root-cause hypothesis.

How I used SigNoz

SRE Sidekick leans on SigNoz across the board — traces, metrics, logs, dashboards, alerts, and the MCP server. Here's how each piece fits.

1. Reproducible install with Foundry

SigNoz + its MCP server come up from a single casting.yaml:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment: { mode: docker, flavor: compose }
  mcp:
    spec:
      enabled: true
Enter fullscreen mode Exit fullscreen mode 
curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml   # SigNoz UI + MCP server, one command
Enter fullscreen mode Exit fullscreen mode

2. The agent's brain and hands

The brain is a Groq-hosted LLM (the gpt-oss family; the provider is swappable via one env
var — Ollama or any hosted model). The hands are the 41 tools the SigNoz MCP server
exposes. My code is the nervous
system connecting them in a loop:

think (LLM picks a tool) → act (call it via MCP) → observe (feed result back) → repeat → answer
Enter fullscreen mode Exit fullscreen mode

Nobody hard-codes the sequence — the LLM decides each step. That autonomy is what makes it an
agent rather than a chatbot.

3. The self-observability loop (my favourite part)

The agent instruments itself with OpenTelemetry and exports to the same SigNoz:

with tracer.start_as_current_span("agent.run"):
    ...
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("gen_ai.usage.input_tokens", in_tok)
        span.set_attribute("gen_ai.usage.output_tokens", out_tok)
        span.set_attribute("gen_ai.usage.cost_usd", cost)
Enter fullscreen mode Exit fullscreen mode

Open SigNoz → Services and sre-sidekick is right there. Open a trace and you see
agent.run → llm.call → tool.signoz_*, each carrying token counts and cost. The agent that
reads observability is itself observable.

I also wired logs through OpenTelemetry so each step logs a line correlated to its trace,
and built a small "Agent Health" dashboard of token usage and cost over time — completing
traces + metrics + logs + dashboards for the agent itself.

4. Acting on SigNoz, safely

Read tools run autonomously; write tools (import dashboard, create alert) pause and wait for
an explicit Approve / Reject from the user before executing. This models responsible agent
autonomy — the agent never silently mutates your monitoring.

5. Alert → auto-diagnose

The alert I create routes to a webhook on the agent's own server. When it fires, the agent
spins up a fresh investigation and posts a root-cause hypothesis — a closed detect→diagnose loop.

What worked, and what fought back

What worked: grounding every answer in real MCP tool calls made the agent reliable even on mid-size model — the intelligence comes from the data, not just the LLM.

What fought back (and what I learned):

  • Too many tools overflow the context. Exposing all 41 MCP tools at once blew the model's request budget (~43k tokens in one call). Curating to ~10 sharp tools and stripping verbose schema descriptions (down to ~1.4k tokens) made it cheaper and better at picking the right tool.

  • The agent answered from memory. With a running conversation it sometimes answered from earlier context instead of querying live data — so it looked like it wasn't using SigNoz at all. Giving each question a fresh context forced a real tool call every time.

  • Wrong-looking numbers. SigNoz returns span durations in nanoseconds and error rate as a
    percentage; my first version mislabelled units. I moved the conversion into code (not the LLM)
    so the numbers are always correct.

  • Complex write APIs. The create_dashboard/create_alert payloads are deep, and the model
    mis-formatted them. I wrapped them in tiny, hard-to-misuse tools that assemble the correct
    payload in Python — a reliability pattern I'll reuse.

  • Shared context bug. The auto-diagnosis first reused the chat agent's history and looped to
    the step limit; giving it a fresh agent fixed it.

Try it / links

Built with Python, OpenTelemetry, the SigNoz MCP server, and Foundry. This project was built
with the assistance of Claude Code (Anthropic) as an AI coding assistant for coding and
documentation; all architecture, testing, and integration were directed and verified by me.

Takeaway

SigNoz gives you the platform and the MCP tools; the fun is in building an agent that wields
them — and, in the spirit of the hackathon, making the agent observe itself. If you can't
observe your agents, you don't own them. SRE Sidekick makes them fully visible.

Top comments (0)