Teaching a Free, Fast LLM to Watch My Cluster With SigNoz — So It Could Tell Me What's Wrong Before I Asked

Monitoring That Doesn't Wait to Be Asked

Most "AI observability" demos wait for you to ask a question. Mine doesn't.
Every five minutes, my agent checks a live SigNoz instance on its own, and
only speaks up when something's actually broken — with a proposed fix
already attached. If you've ever wanted your monitoring to tell you about a
problem before you opened the dashboard, this is the build log: what I made,
what actually broke while building it, and the three categories of pain that
ate almost all my time.

Why SigNoz's MCP Server Was the Right Foundation

I called the agent Sidekick. The goal was narrow on purpose: an agent
that
(1) answers real questions about a real running system — "why is
checkout slow?" — by actually querying live data instead of guessing, and
(2) checks in on its own, unprompted, and only interrupts me when it's
found something worth knowing.

SigNoz was the right backend for one concrete reason: it ships an
official MCP server that exposes traces, metrics, logs, and alerts as
tools an LLM can call directly. That's the difference between "a
dashboard" and something an agent can actually use — the LLM isn't reading
a screenshot, it's calling a tool and getting real numbers back.

Building Sidekick: Data, Bridge, Agent

I deployed SigNoz to a local Kubernetes cluster (kind)
via Helm, then installed SigNoz's k8s-infra chart, which drops an
OpenTelemetry Collector onto every node as a DaemonSet, plus a
cluster-level collector. This piece runs forever, independent of anything
AI-related — it's just continuously feeding SigNoz real metrics, logs, and
traces.

SigNoz's MCP server sits between the data and the agent. I
ran it as a Docker container:

bash command used-

docker run -d --name signoz-mcp -p 8000:8000 \
  -e SIGNOZ_URL="http://host.docker.internal:8080" \
  -e SIGNOZ_API_KEY="<service account key>" \
  -e TRANSPORT_MODE="http" \
  -e MCP_SERVER_PORT="8000" \
  signoz/signoz-mcp-server:latest

Sidekick runs two loops at once. In chat mode, you ask a
question, Groq decides which SigNoz tool to call — search traces, check
alerts, pull metrics — reads the real result, and answers in plain
English. In watch mode, every WATCH_INTERVAL_SECONDS (I used 300),
Groq runs that same investigation on itself, and I force it to end with a
strict JSON verdict:

json
{"status": "ok" | "issue", "service": "...", "summary": "...",
"evidence": "...", "proposed_fix": "..."}

That's the piece that makes "should I interrupt the human" a reliable,
parseable decision instead of a fuzzy paragraph. I also fingerprint each
issue by service + summary, so an ongoing incident doesn't re-alert me
every single cycle — only a genuinely new problem does.

I kept the LLM behind one small interface — send(system, history, tools)
-> AssistantTurn — rather than hard-coding Groq's SDK shape into the chat
loop or the watcher directly. Groq's tool-calling responses come back with
real id fields on each call, so matching a tool result to the call that
requested it is exact, not inferred.

Where It All Actually Broke

Three categories of problems ate almost all
my time — not the AI logic, which mostly worked once it had real data to
look at.

Connection issues, over and over. The MCP server runs inside a Docker
container, and localhost from inside that container means the container
itself, not my host machine — so pointing SIGNOZ_URL at
localhost:8080 silently failed. host.docker.internal was the fix. My
first docker run (no -d) also looked healthy in the logs but was
running in the foreground — closing that terminal killed it, and the only
proof was buried in the logs: "msg":"Received shutdown signal". And
there's no Windows binary for SigNoz's MCP server at all — its build only
targets Linux/macOS — so Docker wasn't a workaround, it was the only path.
The lesson underneath all three: verify the connection independently
(docker ps, docker logs, a plain curl) before assuming the next layer
up is broken.

LLM token and API issues. Groq's free tier meant living inside real rate
limits, not theoretical ones. A single health check making 3–4 tool calls
in a row could burn through the free-tier requests-per-minute limit fast,
especially with chat and the background watcher both hitting the API
around the same time. Some calls also failed outright on bad keys before I
had retries wired up — a 401 gives almost no detail to debug from. Adding
exponential backoff specifically on 429s and 5xxs turned this from "the
watcher silently stops working" into "it waits a few seconds and tries
again."

CPU overload running the whole stack locally. SigNoz isn't one process —
it's SigNoz, ClickHouse, ClickHouse Keeper, and Postgres as containers,
plus a Kubernetes control plane underneath, plus the MCP server, plus
Sidekick itself. On one laptop, that's real concurrent CPU draw, and I
watched it climb close to what Docker Desktop had available — enough that
I had to actually check container CPU/memory usage rather than assume
things were fine just because nothing had crashed yet.

What I'd Tell Myself Before Starting Over

• Check docker ps, not just the logs, before trusting a container is still alive.** Clean startup logs don't mean it's still running five minutes later.
• The AI part is the easy part.** The LLM deciding which tool to call is maybe 20% of the work. The other 80% is plumbing — data flowing, network paths between a container, a cluster, and localhost.
• Forcing a strict JSON verdict was the single best design decision.** Without it, "is this a problem?" is a fuzzy paragraph I'd have to re- read every time.
• Keeping the LLM behind one small interface paid off**, even with a single provider — separating "how I talk to the model" from "what I do with its answer" made chat and the watcher trivial to share.

The Real Payoff

SigNoz's MCP server is what made this feel like a real agent instead of a
wrapper around a dashboard — an LLM that queried actual error rates five
seconds ago, not one describing what they might look like. If you're
building something similar: get the data flowing and the MCP server
reachable first, and prove it with curl before any LLM goes near it —
every bug above was in that plumbing, not in anything Groq did wrong.