Shivanshu

Posted on Jul 26

Helios: Turning SigNoz Telemetry into an On-Call AI Agent

#signoz #opentelemetry #mcp #agents

After a deploy, the question is rarely “do we have dashboards?” — it’s “what actually broke, and what should we do?” Helios is our answer: an AI agent that treats SigNoz as the source of truth, queries it through the SigNoz MCP, and answers like a sharp on-call engineer.

This post walks through what we built for the Agents of SigNoz hackathon, how OpenTelemetry and SigNoz fit in, and the gotchas we hit wiring an LLM to real traces.

What Helios is

Helios is a CLI + optional HTTP API that runs observability tasks:

Command	Job
`helios investigate "…"`	Freeform questions over live telemetry
`helios deploy-health`	Compare a deploy window vs baseline
`helios rca`	Root-cause a timeout / error window
`helios autoscale`	Argue for HPA changes from pressure signals

Under the hood it is a standard agent loop: LLM ↔ tools ↔ validated JSON. The tools are not home-grown scrapers — they are SigNoz MCP tools pointed at a local SigNoz CE stack.

Why SigNoz

We needed three things:

OpenTelemetry-native ingest — demo apps already speak OTLP.
Queryable APIs for an agent — MCP exposes list/search/aggregate without reinventing ClickHouse SQL.
A UI humans trust — every Helios answer can deep-link to the same traces an engineer opens in SigNoz.

Self-hosted SigNoz CE (Docker Compose via Foundry/casting) gave us UI on :8080, OTLP on :4317/:4318, and MCP on :8000/mcp with no SaaS lock-in for the demo.

Architecture (brief)

  loadgen ──► checkout ──► payment ──► inventory     
                    │ OTLP
                    ▼
            otel-collector (kind: observability)
                    │ OTLP → host gateway :4317
                    ▼
         ┌──────────────────────────┐
         │  SigNoz CE (Docker)      │
         │  UI :8080 · OTLP · CH    │
         │  MCP :8000/mcp           │
         └────────────▲─────────────┘
                      │ MCP tools
         ┌────────────┴─────────────┐
         │  Helios (CLI / :9090)    │
         │  LLM + agent loop        │
         └──────────────────────────┘

Pipeline roles

Apps — FastAPI services instrumented with OpenTelemetry (FastAPI + httpx auto-instrumentation).
Collector — Batches traces/metrics/logs and exports to SigNoz.
SigNoz — Stores telemetry; UI for humans; MCP for the agent.
Helios — Plans investigations, calls MCP tools, emits structured results.

How we emit telemetry into SigNoz

Each demo service configures a tracer provider and OTLP gRPC exporter toward the in-cluster collector:

endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317")
resource = Resource.create({"service.name": service_name, ...})
provider = TracerProvider(resource=resource)
exporter = OTLPSpanExporter(endpoint=endpoint, insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()

Faults are injected as Deployment env (ERROR_RATE, BASE_LATENCY_MS, FAIL_DOWNSTREAM, CPU_BURN_MS) so we can replay scenarios: healthy baseline, elevated checkout errors, payment→inventory timeouts, and load pressure.

Gotcha we hit: On Docker Desktop + kind, host.docker.internal for the collector → SigNoz hop preferred IPv6 and broke gRPC. Pinning the exporter to the host-gateway IPv4 (192.168.65.254:4317) fixed “traces missing in ClickHouse.”

How Helios reads SigNoz (MCP)

Helios connects to http://localhost:8000/mcp, lists tools (~40 in our demo), and lets the model call them. The playbook is deliberate:

signoz_list_services — confirm the service exists in the window.
signoz_aggregate_traces — count / p95 with aggregateOn=duration_nano, plus error: true for failures.
signoz_search_traces — sample error spans.
signoz_get_trace_details — when RCA needs the payment→inventory story.

Local casting config enables MCP and impersonation so the agent does not need a client API key; the MCP server holds a server-side credential.

Example freeform investigation:

helios --quiet investigate \
  "Why is payment timing out? Is inventory the failing dependency?" \
  --service payment-service \
  --service inventory-service \
  --namespace demo \
  --start "$START" --end "$END"

Typical answer (paraphrased from a real run): payment is timing out because inventory p95 latency sits around ~3s; payment sees ReadTimeout / HTTP 504 — inventory is the failing dependency even when it doesn’t emit error spans of its own.

That distinction — slow dependency vs erroring dependency — only shows up clearly when the agent can join cross-service traces in SigNoz.

Demo scenarios we validated against SigNoz

Scenario	Fault	What Helios should conclude
Healthy	Reset env	Low/zero errors, stable latency
Deploy degraded	Checkout `ERROR_RATE≈0.25`	Elevated errors vs prior window
Downstream timeout	Payment short timeout + slow inventory	Dependency / inventory
Load pressure	Higher RPS + CPU burn	Scale / pressure recommendation

We scripted one-at-a-time injection (make demo-scenario-*), waited for traffic, then ran investigate so signals don’t mix. Answers deep-link to http://localhost:8080 so judges (and we) can verify in the SigNoz UI.

What surprised us

Empty logs ≠ missing telemetry. Our demos are trace-heavy. Early prompts over-weighted “no logs” as failure. We taught the agent: prefer traces; treat empty logs as normal in the local demo.
Aggregations need explicit fields. signoz_aggregate_traces wants aggregation (and aggregateOn for p95). Vague tool calls burn retries.
Schema discipline matters. The model occasionally returned confidence: "high" or severity: "major". We tightened prompts and added Pydantic coercions so the CLI doesn’t explode mid-demo.
Baselines beat absolute thresholds. “10% errors” means more when the prior window was 0% — Helios compares windows the same way a human would in SigNoz.

Takeaways

SigNoz is not just a dashboard in Helios — it is the memory and toolkit of the agent.
OpenTelemetry + collector + SigNoz CE is enough to stand up a credible multi-service demo on a laptop.
MCP is the right abstraction: the LLM learns tool names, not proprietary SQL.
The best agent answers are the ones you can click into SigNoz and reproduce.

If you’re building AI on top of observability, start with a real backend and a real UI. For us, that backend and UI were SigNoz.

DEV Community