I Already Had Grafana + Loki + Prometheus. Here's What Clicked When I Self-Hosted SigNoz Anyway.

Three Tabs Was My Debugging Workflow. Then One Checkout Failed.

Grafana. Loki. Prometheus.

That stack already ran on my Kubernetes cluster. It was fine for dashboards and alerts.

It was not fine for the question I actually ask at 2am:

Why did this checkout fail — and what log line explains the failing span?

For the SigNoz Early Win, I self-hosted SigNoz on the same cluster, left Grafana/Loki/Prometheus alone, and pointed real OpenTelemetry traffic at SigNoz.

I expected a fourth tab.

I got a shorter path from error span → matching log.

Before / after (same broken checkout)

Step	My old stack	SigNoz
1	Find payment failed in Loki	Open Traces, filter GET /checkout + has_error
2	Guess service + time window	Open the waterfall — payment-gateway owns the duration
3	Jump to Prometheus for error rate	Read the log that already carries the same trace_id
4	Hope a trace backend exists	Done

That four-to-three cut is the whole post. Everything else is how I got there.

Context: Same Cluster, Existing Stack Left Alone

Cluster: Kubernetes v1.30.5, Helm v3.16, SigNoz Helm chart signoz/signoz 0.132.2.

Other workloads already lived there (pipelines, notebooks, a catalog service), plus Grafana + Loki + Prometheus.

No cloud load balancer — UIs are exposed the same way Grafana is: NodePort. SigNoz UI: 32010.

Install: Helm + NodePort (reproducible)

helm repo add signoz https://charts.signoz.io
helm repo update
kubectl create namespace signoz

helm upgrade --install signoz signoz/signoz \
  -n signoz \
  -f helm/signoz-values.yaml \
  --timeout 25m \
  --wait

helm upgrade --install signoz-k8s-infra signoz/k8s-infra \
  -n signoz \
  -f helm/k8s-infra-values.yaml \
  --wait

kubectl apply -f k8s/demo-app.yaml

Or: ./scripts/deploy.sh

Values that mattered for me:

# helm/signoz-values.yaml (excerpt)
global:
  cloud: other
  clusterName: my-cluster

clickhouse:
  persistence:
    enabled: true
    size: 20Gi

signoz:
  service:
    type: NodePort
    port: 8080
    nodePort: 32010 

About five minutes later ClickHouse, ZooKeeper, signoz-0, and the OTel Collector were Running.

Grafana, Loki, and Prometheus stayed untouched.

Coexistence was the point — not a migration fantasy.

Sending Real Data: Break Checkout on Purpose

Demo FastAPI service:

• /inventory, /checkout, metrics probe
• ~25% of checkouts fail with a nested payment-gateway span
• exports traces and logs over OTLP
• CronJob keeps traffic fresh

Gotcha #1: OTLP/gRPC (4317) hit transient UNAVAILABLE right after UI signup. OTLP/HTTP on 4318 worked immediately.

OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector.signoz.svc.cluster.local:4318
# posts to .../v1/traces and .../v1/logs

Then k8s-infra DaemonSets shipped cluster metrics into the same backend.

Same-afternoon evidence

Signal	Count
Spans (signoz-demo-api)	3.6k+
Error spans	270+
Logs	900+
Metric series	7k+

Failing checkout shape:

GET /checkout           hasError=true   ~1203 ms
  checkout              hasError=true
    payment-gateway     hasError=true   ~1201 ms

Matching log (same trace_id):

payment failed after 1200ms

What I Touched (Early Win Checklist)

1. Traces — error list for signoz-demo-api

2. Trace detail — the join starts here

Waterfall for the failing GET /checkout. payment-gateway is the hop that eats the duration and carries the error.

This is the money screenshot: open this span tree, keep the trace_id, then look at logs — you should land on the payment failure without rebuilding the query by hand.

3. Logs — same failure, same trace_id

4–5. Metrics + dashboards

Cluster series from k8s-infra plus spanmetrics-derived latency for the demo service on a small RED-style board. Useful for “is the cluster OK?” next to “is this request OK?”

6. Alerts — checkout-error-spike

Notification channel first (the API refuses rules without one). Rule fired almost immediately — expected for a service that fails on purpose.

Favorite Feature: The Join I Stopped Building by Hand

Grafana + Loki + Prometheus can correlate traces and logs.

In practice my muscle memory was:

1. Spot a failure in logs
2. Guess service + window
3. Open another tool for traces (if one exists)
4. Lose the thread

In SigNoz:

1. Traces → GET /checkout + has_error
2. Waterfall → payment-gateway
3. Log with the same trace_id → payment failed after 1200ms

OpenTelemetry does the correlation when both signals share context. SigNoz’s UI assumes you want that join and makes the click path the default — not a custom derived field I have to remember under pressure.

One sentence for the judges:

SigNoz turned “why did this checkout die?” into a click, not a scavenger hunt across tools I already keep running.

What Broke (Tell Past-Me)

• Finish SigNoz UI signup before trusting OTLP
• Prefer OTLP/HTTP for a first demo
• Services list can look empty while query results already have data
• Alert rules need a notification channel first
• Don’t rip out Grafana/Loki/Prom to “prove” SigNoz — coexistence is the better story

What I’d Do Next

• Push this repo public and pin the chart version in CI
• Add a dashboard screenshot + saved query for the checkout error rate
• Wire a real Slack/webhook channel (not just “it fires in the UI”)
• Try the same join on a multi-service path (not only the demo API)
• Only then talk HA ClickHouse / auth — after the debugging loop is sticky

Thanks for Reading

If you’re doing the Early Win: install SigNoz next to whatever you already run, break one request on purpose, and write about the click that saved you a tab.

Links

• Repo: github.com/Vickyavh7/signoz-k8s-early-win (Helm values, demo app, screenshots)
• SigNoz Early Win hackathon
• Blog guide
• SigNoz Kubernetes install
• k8s-infra docs

---

Comments

[Jishanahmed Shaikh]

Great Read