DEV Community

shahnoor mahesar
shahnoor mahesar

Posted on

Don't Wait for Users to Report Outages: Build a Real-Time Alerting Pipeline with SigNoz

#devops #learning #signoz #opentelemetry

Your service is broken. Somewhere inside a chain of 12 microservices, one call failed silently. Your users see a blank screen. You have no idea where to start looking.

This is exactly what happened when I set up the OpenTelemetry Astronomy Shop demo and intentionally introduced a cascading failure into it. This is also how I learned to use SigNoz, an open-source, self-hostable APM tool, to catch it in real time and fire a Slack alert the moment things went wrong.

This guide walks you through the full journey: running both stacks locally, connecting them over a shared Docker network, exploring SigNoz's dashboards with live traffic, and setting up a proper metric-based alert with a Slack webhook.

By the end, you'll have a working local observability sandbox you can adapt for your own projects too.

Prerequisites

Before starting, make sure you have:

  • Docker Engine 20.10+ with the Docker Compose v2 plugin installed
  • At least 6 GB of RAM available for Docker
  • Git installed
  • The following ports available on your machine:
    • 8080 (SigNoz UI)
    • 4317 (OTLP gRPC ingestion)
    • 4318 (OTLP HTTP ingestion)

Note: If any of these ports are already in use, stop the conflicting service or update your Docker Compose configuration before proceeding.

Architecture

The OpenTelemetry Demo and SigNoz run as separate Docker Compose stacks in different directories, keeping each project isolated and easy to manage.

They communicate over a shared external Docker bridge network named signoz-network. The OpenTelemetry Collector exports telemetry directly to SigNoz using its container name, eliminating the need for localhost or host.docker.internal.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/q4u6elbbd1r4rj5d2n0v.png

Step 1: Running SigNoz Locally with Foundry

The recommended way to install SigNoz on Docker is through Foundry, which is SigNoz's own CLI tool that provisions and manages your self-hosted observability stack. No cloning repos, no copying YAML files manually.

Install foundryctl

curl -fsSL https://signoz.io/foundry.sh | bash
Enter fullscreen mode Exit fullscreen mode

This installs the foundryctl binary to your system. You can verify it worked with:

foundryctl version
Enter fullscreen mode Exit fullscreen mode

Create a project directory and casting file

Create a new directory for your SigNoz setup:

mkdir signoz-local && cd signoz-local
Enter fullscreen mode Exit fullscreen mode

Inside it, create a file named casting.yaml with the following content:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
Enter fullscreen mode Exit fullscreen mode

This declarative config tells Foundry to use Docker Compose as the deployment method.

Deploy SigNoz

Run the cast command:

foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

Foundry validates your Docker setup, generates all the required Compose files into a pours/deployment/ folder, and starts the SigNoz containers automatically. You do not need to run docker compose up yourself.

Wait about 1-2 minutes for ClickHouse and the query service to finish initializing.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/of5hamiw8d1tc2vdfvma.png

Open SigNoz and create your account

Open your browser and go to http://localhost:8080. You'll see the SigNoz onboarding screen. Create your admin account and log in.

Once you're in, your SigNoz instance is ready to receive telemetry data on port 4317 (gRPC) and 4318 (HTTP).

Step 2: Running the OpenTelemetry Demo App

The OpenTelemetry Demo is a realistic microservices app, the "Astronomy Shop," built by the OpenTelemetry community. It comes pre-instrumented with OpenTelemetry SDKs across 20+ services written in Go, Java, Python, .NET, and more. It also ships with a built-in load generator (Locust) that automatically simulates user traffic, so you don't have to do anything to generate telemetry.

Clone the repository

In a separate terminal (not inside the signoz-local folder), clone and enter the demo:

git clone https://github.com/open-telemetry/opentelemetry-demo.git
cd opentelemetry-demo
Enter fullscreen mode Exit fullscreen mode

Fix the port conflict before starting

Both SigNoz and the OTel Demo want port 8080. SigNoz uses it for its web UI. The OTel Demo's Envoy front-end proxy also binds to 8080 by default. If you start both without changing anything, the second one to start will fail with a port allocation error.

The fix is to move the OTel Demo off port 8080. Open the .env file in the root of opentelemetry-demo/ and change these two lines:

# Before
FRONTEND_PORT=8080
ENVOY_PORT=8080

# After
FRONTEND_PORT=3000
ENVOY_PORT=3000
Enter fullscreen mode Exit fullscreen mode

Now the Astronomy Shop will run on http://localhost:3000 and SigNoz stays on http://localhost:8080.

Connect the OTel Demo to SigNoz via a shared network

The OTel Demo's collector needs to send traces and metrics to SigNoz. Since they run in separate Docker Compose stacks, the containers don't share a network by default. Using host.docker.internal is fragile and inconsistent across environments. The clean solution is to join the OTel Demo's collector to SigNoz's existing Docker bridge network.

Open compose.yaml in opentelemetry-demo/. The networks: block already exists at the top of the file. You only need to add the signoz-network entry to it:

networks:
  default:
    name: opentelemetry-demo
    driver: bridge
  # Add this block to join the SigNoz Docker network
  signoz-network:
    name: signoz-network
    external: true
Enter fullscreen mode Exit fullscreen mode

Then, find the otel-collector service definition in the same file and attach it to both networks:

  otel-collector:
    networks:
      - default
      - signoz-network
Enter fullscreen mode Exit fullscreen mode

Point the OTel Collector exporter to SigNoz

Open src/otel-collector/otelcol-config-extras.yml. This file overrides the default collector config. Add the OTLP exporter pointing to SigNoz by its internal container name:

exporters:
  otlp:
    endpoint: "signoz-otel-collector:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      exporters: [spanmetrics, otlp]
    metrics:
      exporters: [otlp]
    logs:
      exporters: [otlp]
Enter fullscreen mode Exit fullscreen mode

Note: The exact container name for the SigNoz collector may vary. Check the names of running SigNoz containers with docker ps to confirm. Look for a container with collector or ingester in its name.

Start the OTel Demo

docker compose up -d
Enter fullscreen mode Exit fullscreen mode

All services will start. The built-in Locust load generator starts automatically and begins sending simulated traffic to the store. Open http://localhost:3000 to see the Astronomy Shop running.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/2xl0hrirjq5m0j3cja9p.png

Step 3: Exploring SigNoz with Live Traffic

Within a minute of the OTel Demo starting, telemetry data will begin flowing into SigNoz. Let's see what we have.

Services Dashboard

Open SigNoz at http://localhost:8080 and click on Services in the left sidebar. You will see every instrumented service from the Astronomy Shop listed here, each showing its live error rate, latency (P50/P99), and request throughput.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/ssyd6jhw5ereyozsvemi.png

Traces Explorer

Click on Traces in the sidebar. SigNoz shows you every distributed trace passing through the system. Click on any individual trace to open the flamegraph, a detailed waterfall chart showing how long each service call took and where latency or errors occurred.

This is where SigNoz really shines. Instead of reading logs line by line, you can visually see the entire request lifecycle from the frontend all the way down to the database layer.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/pkknbb98dnj2804fc6kb.png

Clicking into any trace opens the full flamegraph, showing every service call, its duration, and the exact span attributes collected by OpenTelemetry. You can see the complete request chain from the frontend down to the individual database calls in one view.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/hpyqcndlhl0ugtlv4q8g.png

Step 4: Simulating a Cascading Failure

Watching everything work normally is satisfying, but observability earns its value when things break. The OTel Demo has a built-in feature flag system for exactly this purpose.

Open http://localhost:3000/feature in your browser. You'll see a list of flags you can toggle to inject failures. Turn on productCatalogFailure.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/yqwjgowi03t0b49qyh9d.png

What happens and why it's interesting

Go back to SigNoz and open the Traces tab. Filter by error.type = '500' and you will immediately see which API endpoints are failing. In this case, GET /api/recommendations and GET /api/products/[productId]/index are both returning 500 responses from the frontend service.

This is where SigNoz goes beyond basic log monitoring. Click into any of these failing traces and the flamegraph shows you exactly why the 500 happened: the frontend's API handler called oteldemo.ProductCatalogService/GetProduct internally, and that gRPC call threw an error with the message Error: Product Catalog Fail Feature Flag Enabled. One broken downstream service caused two different API endpoints to fail simultaneously.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/10a1n1lp1vmapvhpt3id.png

The cascade

Because Recommendation Service depends on Product Catalog to fetch product data, it starts failing too. In the error rate chart on the Services dashboard, you'll see multiple services' error rates climb together. A textbook cascading dependency failure, captured and visualized automatically by SigNoz without writing a single additional line of code.

Step 5: Setting Up Slack Alerts

Watching dashboards is not a sustainable on-call strategy. Let's configure SigNoz to automatically alert you in Slack when the error rate crosses a threshold.

Create a Slack Incoming Webhook

In Slack:

  1. Go to Your workspace > Apps > search for Incoming Webhooks
  2. Click Add to Slack and choose a channel (e.g. #alerts)
  3. Copy the generated Webhook URL

Add the Slack channel in SigNoz

In SigNoz, go to Settings > Alert Channels > New Alert Channel. Select Slack, paste the webhook URL, name the channel, and save.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/n06bi9kn9diwbvrcv0lr.png

Test the connection using the "Test" button to confirm a message arrives in your Slack channel.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/jd1erq30rjx86po22gas.png

Create a metric-based alert rule

Go to Alerts > Alert Rules > New Alert Rule > Metric based Alert.

Configure the query as follows:

  • Metric: signoz_calls_total
  • Filter: service.name = 'frontend' AND status.code = 'STATUS_CODE_ERROR'
  • Aggregate within time series: Rate, every 60 Seconds
  • Aggregate across time series: Sum, grouped by service.name and operation

Grouping by service.name and operation is critical. Without it, SigNoz collapses all the data into a single number and drops the label context, which means your alert message will show up with blank service and endpoint fields.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/q0lk3qj1f5wlyqp5ht74.png

Set up the query as shown in the screenshot below:

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/m13junzb3j9eqzigl072.png

Set the alert condition

Scroll down to Set Alert Conditions:

  • Condition: Above (>)
  • Threshold: 0.1 (requests/sec)
  • Evaluation window: Last 5 minutes, Rolling
  • Send to: select your Slack channel

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/ajqadzookhujvnimm477.png

Customize the alert message

Scroll down to Add Details. Give the rule a name (e.g. Frontend Endpoint Error Spike) and paste this into the Description field to get a clean, readable Slack notification:

🔴 *Severity:* {{ $labels.severity }}
📦 *Service:* `{{ $labels.service_name }}`
âš¡ *Endpoint:* `{{ $labels.operation }}`
📈 *Current Error Rate:* `{{ $value }} req/s` (Threshold: `{{ $threshold }} req/s`)

The error rate has exceeded the alert threshold. Review active traces in SigNoz to identify the root cause.
Enter fullscreen mode Exit fullscreen mode

Save the rule. Within 1-2 minutes of productCatalogFailure being enabled, the error rate on the frontend crosses 0.1 req/s, and a fully populated alert fires into your Slack channel.

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/wu1stp0o1zj34itym533.png

Alerts are being fired for the error spikes:

https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/qhvi9hzfuti5vtwuiz7q.png

Conclusion

What we built here: decoupled stacks, shared Docker networking, trace-level failure investigation, and a Slack-integrated alert rule. This is how real production observability setups work, just on a laptop. The tools are open source, self-hostable, and free to try.

If you want to go further, SigNoz also has anomaly detection alerts, SLO/SLA tracking, and infrastructure metrics dashboards. Start with the SigNoz docs and the OpenTelemetry Demo repo if you want to explore from here.

Top comments (0)