Self-hosting SigNoz: notes from a day of actually running it

Last week I self-hosted SigNoz, wired a small app into it, and spent a day poking at traces, logs, dashboards, and alerts. Most of it worked the way the docs said it would. Two things did not, and one of those turned into the most useful thing I learned all day: my logs and my traces were both arriving in SigNoz, but they didn't know about each other.

This post is a writeup of the whole thing, including the parts where I got stuck.

The app

I needed something real to observe, so I built Shelfie, a Flask + SQLite API for tracking books. It has the usual CRUD routes, plus a /books/<id>/recommendations endpoint that calls the Open Library API live. I also added two endpoints on purpose:

• related-slow fetches all the book IDs for a subject, then queries each one individually (the classic N+1 pattern)
• related-fast returns the same data in a single query

The idea was to have a known performance bug in the app and see how obvious SigNoz could make it.

Deploying SigNoz

SigNoz now ships an installer called Foundry:

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml

The casting.yaml is seven lines saying "docker compose, please". That generated and started six containers: ClickHouse, a Postgres metastore, the OTel collector, and the main signoz server.

Two things tripped me up here. First, the UI is on port 8080 now. A lot of older tutorials say 3301. Second, and this one cost me more time: the collector would not accept any telemetry at first. Its logs showed the opamp connection failing with "cannot create agent without orgId". It turns out that until you create the first admin account, there is no organization in the metastore, and the collector has nowhere to attach your data. I registered via the API (POST /api/v1/register) and the error went away immediately. So if your fresh SigNoz install seems to be dropping everything silently, check whether you ever finished signup.

Instrumenting Flask

Traces needed no code changes at all:

opentelemetry-instrument \
  --traces_exporter otlp --metrics_exporter otlp --logs_exporter otlp \
  flask run --port 5000

I added two custom metrics by hand (a counter for books created and a histogram timing the Open Library calls), then ran a small load generator against the app: normal browsing, recommendations, both related endpoints, and a steady trickle of requests for a book ID that doesn't exist, so there would be a real 404 rate.

One installation problem I would point out : I had pinned opentelemetry-distro==0.48b0, and opentelemetry-bootstrap then installed instrumentation packages from 0.65b0. Pip's resolver complained mid-install and the bootstrap failed. Upgrading everything to a matched set (SDK 1.44.0, instrumentation 0.65b0) fixed it. These packages really do need to move together.

A minute or so after starting the load generator, the Services page showed shelfie with a P99 of 1749 ms (the live Open Library calls are slow) and a 0.47% error rate from the deliberate 404s:

The N+1 in a waterfall

The slowest related-slow trace had 10 spans and took 74.27 ms: the Flask handler span, a connect, and then a stack of single-row SELECTs, each with its full SQL in the span attributes. The equivalent related-fast trace had 5 spans and took 7.94 ms. Same response body, about 9x the latency. You can see the problem in the shape of the waterfall before you read anything:

All of those spans came from auto-instrumentation (Flask, SQLAlchemy, requests). I wrote none of them.

The empty Logs tab

Now the part I actually want to tell you about. I opened one of those traces and clicked the Logs tab in the span details panel, expecting to see the request's log lines. Instead: "No logs found for this trace."

The logs were in SigNoz. I could query them in the Logs explorer. But their trace_id and span_id fields were empty, so nothing linked them to the trace. I checked ClickHouse directly to be sure:

trace_id:
span_id:
body:     127.0.0.1 - - [17/Jul/2026 13:28:33] "GET /books/subjects/romance/related-slow HTTP/1.1" 200 -

That log line is Werkzeug's access log, and it explains the whole thing. The access log is written after the request finishes. By that point the span has ended and there is no active trace context left to inject, so the exporter ships the line with no IDs. Everything was "working". The correlation just silently wasn't happening.

The fix was two small changes. I added a normal logger.info(...) call inside the recommendations handler, so at least one log line is emitted while the span is still open, and I set OTEL_PYTHON_LOG_CORRELATION=true so the logging instrumentation stamps the active trace context onto each record. After a restart, the new log line had a real trace_id in ClickHouse, and the same trace's Logs tab in the UI now showed it:

This trace-to-logs link ended up being my favorite feature in SigNoz. Being able to open a slow request and see only that request's log lines removes the usual step of grepping logs around a timestamp and hoping. But the lesson from the empty tab stands: a connected pipeline is not the same as correlated data. If your framework logs after the span closes, you get nothing, and no error tells you so. You have to open a trace and check.

A dashboard and a real alert

I built a small dashboard with two panels: the custom books-created counter, and a spans-per-minute panel filtered to service.name = 'shelfie'. One note on the counter: it's a monotonic sum, and with the default Rate aggregation my one burst of seed inserts was a barely visible blip. Switching the panel to Increase made it readable.

For alerting I wanted to see an actual delivery, not just a rule turning red. So I wrote a 15-line Python HTTP server that appends every POST body to a file, ran it on port 9999, and created a SigNoz Webhook notification channel pointing at it. One Docker detail: from inside the SigNoz container, localhost is the container, not my machine. docker inspect gave me the bridge gateway address (172.19.0.1), and I tested it with a wget from inside the container before configuring anything. The probe showed up in my receiver's log, so I knew the path worked.

Then I made a trace-based alert rule (span count for shelfie above 0 over a 5-minute rolling window, a threshold chosen to fire immediately under load) and waited. About ten minutes later my receiver logged this:

{"receiver":"shelfie-webhook","status":"firing",
 "alerts":[{"labels":{"alertname":"Shelfie traffic alert","severity":"critical"},
 "annotations":{"summary":"...(current value: 32) crosses the threshold (0)",
 "related_traces":"http://localhost:8080/traces-explorer?..."}}]}

And the rule showed Firing in the UI:

The related_traces field in the payload is a nice touch. It's a link that opens the traces explorer already filtered to the service and the exact time window that fired the alert.

Things I'd tell someone doing this tomorrow

Finish the signup before you debug ingestion. No account means no org, and no org means the collector rejects everything with only a vague log line to show for it.

The UI is on port 8080. Ignore anything that says 3301.

Log correlation is effectively opt-in. Auto-instrumentation gives you traces for free, but if all your logs come from an access logger that runs after the span ends, none of them will carry trace IDs. Log from inside the handler and set OTEL_PYTHON_LOG_CORRELATION=true, then verify by opening a trace's Logs tab. Don't just confirm that logs arrive.

Probe your alert channel before you create the rule. Testing the webhook path with one wget from inside the container meant that when the alert didn't arrive instantly, I knew the delay was the evaluation window and not Docker networking.

Keep the OpenTelemetry Python packages on matching versions. Mixed pins fail at install time in a confusing way.

All of this ran on one WSL2 machine in an afternoon: six containers, one Flask app, and a webhook receiver that now has a real alert payload sitting in its log file.

References: SigNoz docs, OpenTelemetry Python, SigNoz alerting