First Hand Guide to Signoz Observability

Our SigNoz Quest Log: What Actually Broke While Adding Observability to our Project (This was the name of our project)

Caution: this is not the official setup guide. For that, go to the SigNoz Docs. This is our field journal from wiring OpenTelemetry into the Project , our FastAPI/LangGraph agentic pentesting scanner, while running a self-hosted SigNoz stack locally.

Shoutout to SigNoz. LessssGooooo.

Quest 1: Why have my containers exited on their own ?

So we had our WSL Running , Docker Desktop was spinning up already , required RAM was free and so was the mentioned ports in the Official Documentation for setting up Signoz locally on our Docker.

Since it was all new to us, We saw the containers running except few , We thought something was off , cause again it was all new to me.

We could see the these two containers: telemetrystore-clickhouse-user-scripts, telemetrystore-migrator gave me exited status. We could not wrap our heads around this.

When we found out stalled containers

Did we miss something ?
Did we skip any step ?
Is our brain just a peanut ?

We did some digging around and found out :

The telemetrystore-clickhouse-user-scripts container runs ClickHouse setup/user scripts. The telemetrystore-migrator container runs database/schema migrations for SigNoz. Once their job is done, they exit. That is expected behavior, especially if the exit code is 0.

Here we also learnt something new , The Docker containers can be of 2 types.

• For long running sessions
• For short transient sessions , mainly used to running some setup scripts for rest of the application. (This was our case).

When we finally saw Signoz working

So Problem 1 fixed and our brains were not peanuts. Hence Proved.

Quest 2: We wanted to add Redis Telemetry onto our Platform.

The documentation's pre-requisite said we needed to install this first : OpenTelemetry (OTEL) Collector.

OTEL Collector is basically the middleman.The collector talks to Redis, picks up signals like memory usage, connected clients etc and sends it to Signoz.

So we went ahead and installed it. Thought we are done , but no.

The collector was running separately, and so was the Redis.
So when we gave Redis endpoint as localhost:6379, the collector tried to find Redis inside its own container, not inside our actual Redis container.

Our current flow : Redis → OpenTelemetry Collector → SigNoz

So we changed this localhost:6379 to redis:6379 in redis-metrics-collection-config.yaml . Now everything was communicating and things worked brilliantly.

Quest 3: App Telemetry Was Not Enough

Once traces were flowing, we wanted metrics for the rest of the project component , since our project dealt with scanning and analysis via many open-source tools and LLM.

A slow scan could be an LLM issue, a bad DB query, Redis pressure, or one tool container quietly eating CPU.

We used a dedicated otel-collector for Postgres and Redis:

postgresql receiver -> Postgres metrics
redis receiver      -> Redis metrics
filelog receiver    -> Postgres/Redis logs

Then we added a separate docker-stats-collector for per-container CPU, memory, and network usage across scanner-tool containers plus core, worker, postgres, and redis.

We kept it separate because only that collector needed Docker socket access. We had already learned that widening permissions casually is how future-you starts judging past-you.

Traces tells us what happened during this request or scan?
Infra metrics tells us was the machine/container/database healthy while it happened?

We need both, otherwise it will all feel like guess work.

Quest 4: Dashboards Became Our Control Room

Yes , we felt like operating a high stake mission straight from laptops
We created six importable SigNoz dashboard JSON files in:

observability/dashboards/

They covered Scan Pipeline Health, LLM API calls , Operations Overview, Scanner Container Resources, LangGraph/LLM Overview, Postgres, and Redis.

The dashboards tracked things like per-node p95 latency, execution funnel, error rate, token usage by provider,Redis health, Postgres health, and SSE reconnects.

But we did not trust labels blindly. For traces, llm.provider worked well. For our metric project.llm.fallback.count, the provider label came back as:

"provider":"unknown"

So we used trace-level llm.provider for provider comparison, and grouped fallback metrics by agent instead.

One of our Dashboards

FAQ For New SigNoz Folks

• What is the OTEL Collector? Think of it as the middleman. Apps, databases, or receivers send telemetry to it, and it forwards that data to SigNoz.
• Why does localhost keep betraying people in Docker? Because inside a container, localhost means that container itself. Use the service name on the Docker network, like redis:6379 or signoz-ingester:4317.
• Do I need ClickHouse knowledge to use SigNoz? Not for normal use. But when you are verifying whether data really landed, knowing how to query SigNoz's ClickHouse store is extremely useful.
• Are dashboards enough? No. Dashboards are for looking. Alerts are for being woken up before users notice.
• Biggest advice? Do not stop at “the app did not crash”. Prove that the telemetry reached SigNoz. Query it, dashboard it, alert on it. Then sleep.