DEV Community: Samuel Dahunsi

Building a Drop-in SLO & Error-Budget Pack for SigNoz Across Traces, Metrics, and Logs

Samuel Dahunsi — Sun, 26 Jul 2026 10:44:58 +0000

How I built a production-grade observability pack with multi-window burn rate alerts, custom ClickHouse dashboards, and automated fault-injection drills for the Agents of SigNoz Hackathon.

The Hook: Why Most SLO Setup Fails in Practice

Setting up Service Level Objectives (SLOs) sounds simple on paper: pick a target like 99.9% availability, track errors, and stop shipping features when your budget burns out.

In reality, most teams hit a wall. Raw APM tools show request counts and average latencies, but they don’t tell you how fast you are consuming your monthly error budget or whether a 5-minute fault requires waking up an engineer on call. Worse, configuring multi-window burn rate alerts across three separate telemetry signals (Traces, Metrics, and Logs) usually takes days of custom ClickHouse query tuning and YAML wrangling.

For the Agents of SigNoz Hackathon (Track 02: Signals & Dashboards), I wanted to solve this permanently by building the SigNoz SLO & Error-Budget Pack: an all-in-one, drop-in observability suite that instruments Node.js/Express applications across all three OpenTelemetry signals and automatically ships three production-grade dashboards and multi-window burn alerts into SigNoz.

Here is how I built it, the mathematical decisions behind the error budget burn rates, and how to test it end-to-end with live-fire fault injection drills.

Architecture: The 3-Signal Telemetry Pipeline

To make SLO calculation accurate, relying on just one signal is a mistake.

Metrics provide high-frequency RED (Rate, Errors, Duration) counters without sampling overhead.
Traces expose p99 latency and child-span performance across downstream API calls.
Logs provide human-readable diagnostic context when an incident occurs.

Here is how data flows through the stack:

                  ┌─────────────────────────────────────────┐
                  │          demo-service (Express)         │
                  │  - tracing.js (OTel SDK)                │
                  │  - RED Middleware (http_requests_total) │
                  │  - Winston Logger (JSON + trace_id)     │
                  └────────────────────┬────────────────────┘
                                       │ OTLP / gRPC & HTTP
                                       ▼
                  ┌─────────────────────────────────────────┐
                  │       SigNoz OpenTelemetry Collector    │
                  └────────────────────┬────────────────────┘
                                       │ ClickHouse Tables
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             SigNoz Engine                                   │
│  ┌───────────────────────┐ ┌────────────────────────┐ ┌──────────────────┐  │
│  │ Service Health        │ │ SLO & Error Budget     │ │ Cost Efficiency  │  │
│  │ Dashboard             │ │ Dashboard              │ │ Dashboard        │  │
│  └───────────────────────┘ └────────────────────────┘ └──────────────────┘  │
│                                      │                                      │
│                                      ▼                                      │
│                    Multi-Window Alert Rules (API v2)                        │
│                                      │ Webhook Payload                      │
│                                      ▼                                      │
│                     Webhook Catcher Service (:3002)                         │
└─────────────────────────────────────────────────────────────────────────────┘

1. Instrumenting the Three Signals

A. RED Metrics Middleware

Rather than relying solely on trace sampling for request rates, I implemented custom RED metrics in metrics.js using @opentelemetry/api-metrics. Every HTTP request is captured by an Express middleware:

// demo-service/index.js
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    recordRequest({
      route: req.path,
      method: req.method,
      statusCode: res.statusCode,
      durationMs: Date.now() - start,
    });
  });
  next();
});

This records three crucial series into SigNoz’s signoz_metrics.distributed_time_series_v4 table:

http_requests_total: Counter partitioned by route, method, and status_class (2xx, 4xx, 5xx).
http_request_duration: Histogram tracking bucketed latency distributions.
downstream_errors_total: Counter tracking third-party API dependencies (catfact, dogimg, joke).

B. Nested Traces with Downstream Attributes

For distributed tracing, the /data endpoint fans out requests to multiple external APIs in parallel. To observe downstream degradation without breaking root trace spans, each downstream call is wrapped in a explicit child span:

async function fetchOne({ name, url }) {
  const span = trace.getTracer('slo-pack-demo').startSpan(`downstream ${name}`);
  const t0 = Date.now();
  try {
    const resp = await axios.get(url, { timeout: 5000 });
    span.setAttributes({
      'downstream.name': name,
      'http.response.status_code': resp.status,
      'downstream.duration_ms': Date.now() - t0,
    });
    return { name, status: resp.status };
  } catch (err) {
    recordDownstreamError(name);
    span.setAttributes({
      'downstream.name': name,
      'error': true,
    });
    throw err;
  } finally {
    span.end();
  }
}

C. 100% Trace-Correlated Logs

One of the biggest friction points in debugging alerts is jumping from a metrics spike to log output. By configuring winston alongside @opentelemetry/instrumentation-winston, trace context is automatically injected into every stdout log line:

{
  "level": "error",
  "message": "handling /error -> 500",
  "service": "slo-pack-demo",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

In the SigNoz UI, clicking any log entry instantly renders the exact visual trace waterfall.

2. Deriving the SLO & Burn Rate Mathematics

Google’s SRE Handbook recommends multi-window, multi-threshold alerting to avoid alert fatigue. Instead of triggering an alert the second error rate hits 0.1%, we measure how fast the error budget is burning.

For a 30-day SLO window with a 99.9% Availability Target:
$$\text{Allowed Unreliability } (U) = 1 - 0.999 = 0.001 \quad (0.1\%)$$

Fast-Burn Alert (Critical)

Goal: Catch massive outages that would consume 2% of the total monthly error budget in 1 hour.
Burn Rate Formula: $$\text{Burn Rate} = \frac{\text{Budget Consumed}}{\text{Time Ratio}} = \frac{0.02}{1 \text{ hour} / 720 \text{ hours}} = 14.4$$
Condition: If the current 1-hour burn rate exceeds 14.4x, trigger a Critical Alert immediately.

Slow-Burn Alert (Warning)

Goal: Catch persistent, subtle error leaks that consume 5% of the error budget over 6 hours.
Burn Rate Formula: $$\text{Burn Rate} = \frac{0.05}{6 \text{ hours} / 720 \text{ hours}} = 6.0$$
Condition: If the 6-hour burn rate exceeds 6.0x, trigger a Warning Alert.

These exact mathematical expressions were converted into SigNoz API v2 alert definitions under alerts/fast-burn.json, alerts/slow-burn.json, and alerts/latency-slo.json.

3. The 3 Custom Dashboards

The pack includes three pre-configured dashboards in dashboards/:

1. Service Health Dashboard

Request Throughput: Real-time req/sec grouped by status class (2xx vs 5xx).
Latency Percentiles: p50, p90, and p99 derived directly from trace span durations.
Downstream Error Tracker: Isolates third-party API failures from internal app errors.

2. SLO / Error Budget Dashboard

Availability SLI %: Computed in real-time as (1 - (sum(5xx) / sum(total))) * 100.
Remaining Error Budget %: Gauge chart highlighting budget degradation before breach.
Burn Rate Gauges: Live 14.4x fast-burn and 6.0x slow-burn indicators.
Latency SLI %: Percentage of requests completing under the 6-second p99 ceiling.

3. Cost-Efficiency & Waste Dashboard

Waste Ratio %: Quantifies wasted compute spent servicing HTTP 5xx responses.
Waste Index: A hybrid metric ($Latency \times ErrorRatio$) pointing out high-latency, error-prone endpoints that consume disproportionate server resources.

4. Automated Verification & Live-Fire Alert Drills

Automated Test Harness

To ensure the pack remains reliable across environments, I wrote a verification script (scripts/verify-p1.sh) that queries both the SigNoz REST API and ClickHouse database:

SIGNOZ_EMAIL="dahunsisamuel1st@gmail.com" SIGNOZ_PASSWORD="..." ./scripts/verify-p1.sh

Output:

== A. Stack & reproducibility (PRD §7.6)
PASS  A1   SigNoz UI + API healthy
PASS  A2   OTLP HTTP ingest endpoint accepting
PASS  A3   demo service /healthz responds
PASS  A4   webhook catcher up
PASS  A5   casting.yaml + lock committed
== B. All three signals w/ required shape
PASS  B1   traces landing (fresh <2m)
PASS  B2   downstream child spans carry custom attrs
PASS  B3   http_requests_total with route/status_class labels
PASS  B4   http_request_duration histogram series present
PASS  B5   downstream_errors_total with downstream label
PASS  B6   metric samples fresh (<2m)
PASS  B7   logs landing (fresh <2m)
PASS  B8   >=95% of app logs trace-correlated
== C. Alert pack
PASS  C1   webhook notification channel registered
PASS  C2   rule 'SLO fast burn' exists w/ severity=critical
PASS  C3   rule 'SLO slow burn' exists w/ severity=warning
PASS  C4   rule 'Latency SLO breach' exists w/ severity=warning
PASS  C5   no rule stuck without state
== D. Paper requirements
PASS  D1   AI_USAGE.md present
PASS  D2   alert payloads committed
PASS  D3   SLO math documented

passed: 21  failed: 0  skipped-sections: 0

Live-Fire Fault Injection

To test alerting end-to-end without waiting for natural outages, I built environment knobs into the containerized service:

# Inject 90% error rate — forces burn rate to ~18x
cd demo-service
ERROR_RATE=0.9 docker compose up -d demo-service

Within minutes:

The Availability SLI dropped on the SLO Dashboard.
The SLO Fast-Burn Rule turned red (FIRING) in SigNoz.
The standalone Webhook Catcher (http://localhost:3002/) logged the incoming incident JSON payload:

{
  "status": "firing",
  "alerts": [{
    "labels": {
      "alertname": "SLO fast burn — availability error budget",
      "severity": "critical",
      "service": "slo-pack-demo"
    },
    "annotations": {
      "summary": "High availability error budget burn rate detected"
    }
  }]
}

Resetting the container restores normal operation immediately:

ERROR_RATE=0.1 SLOW_MAX_MS=3000 docker compose up -d demo-service

Key Lessons Learned & Gotchas

ClickHouse JSON Ingestion Behavior: When querying signoz_logs.distributed_logs_v2, resources_string['service.name'] must match exact string casing. Using LOWER() or un-indexed metadata queries can degrade query times during dashboard renders.
Bash Execution & curl -sf: When automating API provisioning scripts under set -e, avoid using curl -sf on authentication endpoints. Standard HTTP 401 responses will cause curl to exit non-zero silently without outputting SigNoz's JSON error response.
Trace-Log Context Ingestion: OpenTelemetry Winston instrumentation requires standard stdout JSON formatting to reliably extract trace_id and span_id.

Conclusion & Links

Building this pack demonstrated how powerful OpenTelemetry and SigNoz are when all three signals work in unison. Instead of staring at disconnected graphs during an incident, engineering teams get a single pane of glass showing budget remaining, exact firing alerts, and trace-correlated logs within seconds.

GitHub Repository: signoz-slo-pack
SigNoz Documentation: https://signoz.io/docs
OpenTelemetry Documentation: https://opentelemetry.io/docs

What a Corrupted ClickHouse Binary Taught Me About Self-Hosting SigNoz

Samuel Dahunsi — Thu, 16 Jul 2026 15:58:33 +0000

Three crash-looping containers, one corrupted Docker image, and a trace waterfall that made the entire detour worthwhile.

My SigNoz containers kept crash-looping with CANNOT_PARSE_ELF and Segmentation fault. The install guide doesn't mention either error, because neither is SigNoz's fault. Both are what happens when your machine dies mid docker pull. This is the full trail: what broke, why, and how I got traces flowing anyway.

Why I was doing this

I wanted to actually understand observability, not just read about it. The plan was simple: self-host SigNoz, point a real app at it, and poke at traces, logs, and dashboards until the concepts clicked.

Quick vocabulary, since these words get thrown around a lot:

Telemetry is the data an app emits about what it's doing while it runs, essentially a flight recorder for software.
It comes in three flavors: traces (the journey of one request through your system), metrics (numbers over time, like requests/sec), and logs (the text your app prints).
OpenTelemetry (OTel) is the vendor-neutral standard for formatting and shipping that data.
SigNoz receives all of it, stores it, and gives you dashboards, search, and alerts so you can make sense of it.

My setup: a Mac, Docker Desktop, and SigNoz's newer installer, Foundry.

The clean install (this part is easy)

SigNoz has moved off the old docker-compose instructions to a CLI called Foundry. Three steps:

# 1. Install the Foundry CLI (I downloaded and read it before running it)
curl -fsSL https://signoz.io/foundry.sh -o foundry.sh
bash foundry.sh

# 2. A tiny config file, casting.yaml
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

# 3. Deploy
foundryctl cast -f casting.yaml

That generates a real Compose file under pours/deployment/ and starts every container SigNoz needs: ClickHouse (the database), ClickHouse Keeper (coordination), Postgres (metadata), an OTel Collector (the thing your app sends data to), and the SigNoz app itself.

That part is in the docs. Here's the part that isn't.

Where it broke

My first deploy got interrupted while Docker was still pulling images. When I brought the stack back up, three containers went into a crash-loop. The logs:

Code: 464. DB::Exception: The ELF '/usr/bin/clickhouse' doesn't have
string table with section names. (CANNOT_PARSE_ELF)

Segmentation fault
Segmentation fault
Segmentation fault

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty

Three different services, three different errors. It looked like SigNoz was badly broken. It wasn't.

The diagnosis

The common thread is that all three failures are about half-written files.

Docker Desktop on a Mac doesn't run containers directly on macOS. It runs a hidden Linux VM, and every image, container, and database volume lives on one big virtual disk inside that VM. When Docker (or the whole machine) is killed abruptly instead of shut down cleanly, whatever was mid-write gets left truncated, the same way a file can end up corrupted if you yank a USB stick out while it's still saving.

Mapping the errors back:

Error	What actually happened
`CANNOT_PARSE_ELF`	The ClickHouse binary inside the image was written only partway, so Linux can't parse it as an ELF file
`Segmentation fault` (collector)	Same story: an incomplete binary crashing the instant it runs
Postgres `directory exists but is not empty`	Postgres got interrupted mid-initialization and left a half-built data folder

None of this is a SigNoz bug. It's what an interrupted write looks like at three different layers of the stack.

The fix

The recovery recipe is the same every time: delete the corrupted artifact and recreate it from scratch.

# Stop everything
docker compose down

# Remove the corrupted images so they re-pull cleanly
docker rmi -f clickhouse/clickhouse-server:25.12.5
docker rmi -f signoz/signoz-otel-collector:latest

# Remove the half-initialized Postgres volume
docker volume rm signoz-metastore-postgres-0-data

# Re-pull and verify the binary is actually intact this time
docker compose pull
docker run --rm --entrypoint clickhouse \
  clickhouse/clickhouse-server:25.12.5 local --version
# -> ClickHouse local version 25.12.5.44 (official build)   ✅

# Bring it back up
docker compose up -d

That last verify step is the one I'd skip in a hurry, and the one I'd regret skipping. Checking the binary before starting the stack turns a 20-minute crash-loop mystery into a five-second yes/no.

After that, every container came up healthy:

signoz-signoz-0                            Up (healthy)
signoz-telemetrystore-clickhouse-0-0       Up (healthy)
signoz-metastore-postgres-0                Up (healthy)
signoz-telemetrykeeper-clickhousekeeper-0  Up (healthy)
signoz-ingester-1                          Up

SigNoz UI came up at http://localhost:8080.

the SigNoz home dashboard showing traces and metrics ingestion both active

Getting real data in

An empty observability tool is useless, so I grabbed SigNoz's Node.js sample app, which is already OpenTelemetry-instrumented. It has one endpoint, /data, that calls three public APIs in parallel: a cat fact, a dog photo, and a joke.

The only change needed was pointing it at my local collector instead of SigNoz Cloud, in tracing.js. Hitting the endpoint directly confirms what it returns:

the /data endpoint's JSON response showing catFact, dogImage, and joke fields

const exporterOptions = {
  url: 'http://host.docker.internal:4318/v1/traces',
};

host.docker.internal is how a container reaches a service running on the host machine, in this case the OTel Collector listening on port 4318.

One gotcha worth saving you the hour I lost: on my network, the app's outbound calls kept failing with connect ENETUNREACH because Node was resolving the APIs to IPv6 addresses my router wasn't routing. The fix was one line in the Compose file to prefer IPv4:

environment:
  - NODE_OPTIONS=--dns-result-order=ipv4first

Then a build, and a burst of traffic:

docker compose up -d --build
for i in $(seq 1 20); do curl -s http://localhost:3000/data; done

The feature I liked most: the trace waterfall

This is where tracing earned its keep. A burst of traffic against /data lands as a flat list first:

the Traces Explorer list view showing spans from a burst of traffic against /data

Click into one trace and the /data endpoint fires three API calls at once with Promise.all, and the trace waterfall shows exactly that: a parent GET /data span with three child spans running side by side.

the /data trace waterfall showing a parent span with three parallel child spans

The insight lands in one glance. Because the three calls are parallel, the total response time isn't their sum, it's whichever single call was slowest. When a request was slow, I could see which of the three APIs dragged it, not guess. That's the whole pitch of distributed tracing, made concrete in one screen.

The part that genuinely surprised me: I wrote zero tracing code to get this. No manual spans, no timers. getNodeAutoInstrumentations() wrapped Express and axios automatically and produced the entire waterfall, including each outbound call's URL, method, and status code as span attributes.

a single GET span expanded, showing auto-captured HTTP attributes like http.method, http.url, and http.status_code

What worked, what didn't

Worked: traces, immediately and impressively. 218 spans from one burst of traffic, waterfalls rendering with parallel child spans and full HTTP attributes, no custom instrumentation code.
Also worked: metrics. http.client.duration.bucket and http.server.duration.bucket both showed up in the Metrics Explorer with thousands of samples, no extra config beyond what Foundry sets up by default.

[the Metrics Explorer showing http.client.duration.bucket and http.server.duration.bucket with 12.1K+ and 3.2K+ samples]

Didn't (yet): logs. I wired up an OTel log exporter too, but it didn't land in SigNoz on the first pass while traces and metrics did. Traces, logs, and metrics are three separate export pipelines, and getting two working doesn't mean the third is, a lesson I'd rather learn on a toy app than in production. That's my next debugging session.
The big one: an abrupt shutdown can corrupt Docker's VM disk, and the fix is always the same: delete the broken image or volume and recreate it. If your machine has ever hung and forced you to hard-quit, this is worth filing away for next time.

Conclusion

The install is genuinely three commands. The real learning was in the failure: reading CANNOT_PARSE_ELF, a segfault, and a Postgres init error, and realizing they were all the same problem wearing different masks. Traces made my toy app's behavior obvious at a glance, and I got there without writing a line of instrumentation. Metrics came through cleanly too. Logs are the unfinished piece, which feels like the right note to end a first-week-with-a-new-tool post on.

Links:

SigNoz docs: https://signoz.io/docs
Foundry install guide: https://signoz.io/docs/install/docker/
OpenTelemetry: https://opentelemetry.io
SigNoz Node.js sample app: https://github.com/SigNoz/sample-nodejs-app