DEV Community

Samuel Dahunsi
Samuel Dahunsi

Posted on

Building a Drop-in SLO & Error-Budget Pack for SigNoz Across Traces, Metrics, and Logs

#programming #signoz #agents

How I built a production-grade observability pack with multi-window burn rate alerts, custom ClickHouse dashboards, and automated fault-injection drills for the Agents of SigNoz Hackathon.

The Hook: Why Most SLO Setup Fails in Practice

Setting up Service Level Objectives (SLOs) sounds simple on paper: pick a target like 99.9% availability, track errors, and stop shipping features when your budget burns out.

In reality, most teams hit a wall. Raw APM tools show request counts and average latencies, but they don’t tell you how fast you are consuming your monthly error budget or whether a 5-minute fault requires waking up an engineer on call. Worse, configuring multi-window burn rate alerts across three separate telemetry signals (Traces, Metrics, and Logs) usually takes days of custom ClickHouse query tuning and YAML wrangling.

For the Agents of SigNoz Hackathon (Track 02: Signals & Dashboards), I wanted to solve this permanently by building the SigNoz SLO & Error-Budget Pack: an all-in-one, drop-in observability suite that instruments Node.js/Express applications across all three OpenTelemetry signals and automatically ships three production-grade dashboards and multi-window burn alerts into SigNoz.

Here is how I built it, the mathematical decisions behind the error budget burn rates, and how to test it end-to-end with live-fire fault injection drills.

Architecture: The 3-Signal Telemetry Pipeline

To make SLO calculation accurate, relying on just one signal is a mistake.

  • Metrics provide high-frequency RED (Rate, Errors, Duration) counters without sampling overhead.
  • Traces expose p99 latency and child-span performance across downstream API calls.
  • Logs provide human-readable diagnostic context when an incident occurs.

Here is how data flows through the stack:

                  ┌─────────────────────────────────────────┐
                  │          demo-service (Express)         │
                  │  - tracing.js (OTel SDK)                │
                  │  - RED Middleware (http_requests_total) │
                  │  - Winston Logger (JSON + trace_id)     │
                  └────────────────────┬────────────────────┘
                                       │ OTLP / gRPC & HTTP
                                       ▼
                  ┌─────────────────────────────────────────┐
                  │       SigNoz OpenTelemetry Collector    │
                  └────────────────────┬────────────────────┘
                                       │ ClickHouse Tables
                                       ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             SigNoz Engine                                   │
│  ┌───────────────────────┐ ┌────────────────────────┐ ┌──────────────────┐  │
│  │ Service Health        │ │ SLO & Error Budget     │ │ Cost Efficiency  │  │
│  │ Dashboard             │ │ Dashboard              │ │ Dashboard        │  │
│  └───────────────────────┘ └────────────────────────┘ └──────────────────┘  │
│                                      │                                      │
│                                      ▼                                      │
│                    Multi-Window Alert Rules (API v2)                        │
│                                      │ Webhook Payload                      │
│                                      ▼                                      │
│                     Webhook Catcher Service (:3002)                         │
└─────────────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

1. Instrumenting the Three Signals

A. RED Metrics Middleware

Rather than relying solely on trace sampling for request rates, I implemented custom RED metrics in metrics.js using @opentelemetry/api-metrics. Every HTTP request is captured by an Express middleware:

// demo-service/index.js
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    recordRequest({
      route: req.path,
      method: req.method,
      statusCode: res.statusCode,
      durationMs: Date.now() - start,
    });
  });
  next();
});
Enter fullscreen mode Exit fullscreen mode

This records three crucial series into SigNoz’s signoz_metrics.distributed_time_series_v4 table:

  • http_requests_total: Counter partitioned by route, method, and status_class (2xx, 4xx, 5xx).
  • http_request_duration: Histogram tracking bucketed latency distributions.
  • downstream_errors_total: Counter tracking third-party API dependencies (catfact, dogimg, joke).

B. Nested Traces with Downstream Attributes

For distributed tracing, the /data endpoint fans out requests to multiple external APIs in parallel. To observe downstream degradation without breaking root trace spans, each downstream call is wrapped in a explicit child span:

async function fetchOne({ name, url }) {
  const span = trace.getTracer('slo-pack-demo').startSpan(`downstream ${name}`);
  const t0 = Date.now();
  try {
    const resp = await axios.get(url, { timeout: 5000 });
    span.setAttributes({
      'downstream.name': name,
      'http.response.status_code': resp.status,
      'downstream.duration_ms': Date.now() - t0,
    });
    return { name, status: resp.status };
  } catch (err) {
    recordDownstreamError(name);
    span.setAttributes({
      'downstream.name': name,
      'error': true,
    });
    throw err;
  } finally {
    span.end();
  }
}
Enter fullscreen mode Exit fullscreen mode

C. 100% Trace-Correlated Logs

One of the biggest friction points in debugging alerts is jumping from a metrics spike to log output. By configuring winston alongside @opentelemetry/instrumentation-winston, trace context is automatically injected into every stdout log line:

{
  "level": "error",
  "message": "handling /error -> 500",
  "service": "slo-pack-demo",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}
Enter fullscreen mode Exit fullscreen mode

In the SigNoz UI, clicking any log entry instantly renders the exact visual trace waterfall.

2. Deriving the SLO & Burn Rate Mathematics

Google’s SRE Handbook recommends multi-window, multi-threshold alerting to avoid alert fatigue. Instead of triggering an alert the second error rate hits 0.1%, we measure how fast the error budget is burning.

For a 30-day SLO window with a 99.9% Availability Target:
$$\text{Allowed Unreliability } (U) = 1 - 0.999 = 0.001 \quad (0.1\%)$$

Fast-Burn Alert (Critical)

  • Goal: Catch massive outages that would consume 2% of the total monthly error budget in 1 hour.
  • Burn Rate Formula: $$\text{Burn Rate} = \frac{\text{Budget Consumed}}{\text{Time Ratio}} = \frac{0.02}{1 \text{ hour} / 720 \text{ hours}} = 14.4$$
  • Condition: If the current 1-hour burn rate exceeds 14.4x, trigger a Critical Alert immediately.

Slow-Burn Alert (Warning)

  • Goal: Catch persistent, subtle error leaks that consume 5% of the error budget over 6 hours.
  • Burn Rate Formula: $$\text{Burn Rate} = \frac{0.05}{6 \text{ hours} / 720 \text{ hours}} = 6.0$$
  • Condition: If the 6-hour burn rate exceeds 6.0x, trigger a Warning Alert.

These exact mathematical expressions were converted into SigNoz API v2 alert definitions under alerts/fast-burn.json, alerts/slow-burn.json, and alerts/latency-slo.json.

3. The 3 Custom Dashboards

The pack includes three pre-configured dashboards in dashboards/:

1. Service Health Dashboard

  • Request Throughput: Real-time req/sec grouped by status class (2xx vs 5xx).
  • Latency Percentiles: p50, p90, and p99 derived directly from trace span durations.
  • Downstream Error Tracker: Isolates third-party API failures from internal app errors.

2. SLO / Error Budget Dashboard

  • Availability SLI %: Computed in real-time as (1 - (sum(5xx) / sum(total))) * 100.
  • Remaining Error Budget %: Gauge chart highlighting budget degradation before breach.
  • Burn Rate Gauges: Live 14.4x fast-burn and 6.0x slow-burn indicators.
  • Latency SLI %: Percentage of requests completing under the 6-second p99 ceiling.

3. Cost-Efficiency & Waste Dashboard

  • Waste Ratio %: Quantifies wasted compute spent servicing HTTP 5xx responses.
  • Waste Index: A hybrid metric ($Latency \times ErrorRatio$) pointing out high-latency, error-prone endpoints that consume disproportionate server resources.

4. Automated Verification & Live-Fire Alert Drills

Automated Test Harness

To ensure the pack remains reliable across environments, I wrote a verification script (scripts/verify-p1.sh) that queries both the SigNoz REST API and ClickHouse database:

SIGNOZ_EMAIL="dahunsisamuel1st@gmail.com" SIGNOZ_PASSWORD="..." ./scripts/verify-p1.sh
Enter fullscreen mode Exit fullscreen mode

Output:

== A. Stack & reproducibility (PRD §7.6)
PASS  A1   SigNoz UI + API healthy
PASS  A2   OTLP HTTP ingest endpoint accepting
PASS  A3   demo service /healthz responds
PASS  A4   webhook catcher up
PASS  A5   casting.yaml + lock committed
== B. All three signals w/ required shape
PASS  B1   traces landing (fresh <2m)
PASS  B2   downstream child spans carry custom attrs
PASS  B3   http_requests_total with route/status_class labels
PASS  B4   http_request_duration histogram series present
PASS  B5   downstream_errors_total with downstream label
PASS  B6   metric samples fresh (<2m)
PASS  B7   logs landing (fresh <2m)
PASS  B8   >=95% of app logs trace-correlated
== C. Alert pack
PASS  C1   webhook notification channel registered
PASS  C2   rule 'SLO fast burn' exists w/ severity=critical
PASS  C3   rule 'SLO slow burn' exists w/ severity=warning
PASS  C4   rule 'Latency SLO breach' exists w/ severity=warning
PASS  C5   no rule stuck without state
== D. Paper requirements
PASS  D1   AI_USAGE.md present
PASS  D2   alert payloads committed
PASS  D3   SLO math documented

passed: 21  failed: 0  skipped-sections: 0
Enter fullscreen mode Exit fullscreen mode

Live-Fire Fault Injection

To test alerting end-to-end without waiting for natural outages, I built environment knobs into the containerized service:

# Inject 90% error rate — forces burn rate to ~18x
cd demo-service
ERROR_RATE=0.9 docker compose up -d demo-service
Enter fullscreen mode Exit fullscreen mode

Within minutes:

  1. The Availability SLI dropped on the SLO Dashboard.
  2. The SLO Fast-Burn Rule turned red (FIRING) in SigNoz.
  3. The standalone Webhook Catcher (http://localhost:3002/) logged the incoming incident JSON payload: 
{
  "status": "firing",
  "alerts": [{
    "labels": {
      "alertname": "SLO fast burn — availability error budget",
      "severity": "critical",
      "service": "slo-pack-demo"
    },
    "annotations": {
      "summary": "High availability error budget burn rate detected"
    }
  }]
}
Enter fullscreen mode Exit fullscreen mode

Resetting the container restores normal operation immediately:

ERROR_RATE=0.1 SLOW_MAX_MS=3000 docker compose up -d demo-service
Enter fullscreen mode Exit fullscreen mode

Key Lessons Learned & Gotchas

  1. ClickHouse JSON Ingestion Behavior: When querying signoz_logs.distributed_logs_v2, resources_string['service.name'] must match exact string casing. Using LOWER() or un-indexed metadata queries can degrade query times during dashboard renders.
  2. Bash Execution & curl -sf: When automating API provisioning scripts under set -e, avoid using curl -sf on authentication endpoints. Standard HTTP 401 responses will cause curl to exit non-zero silently without outputting SigNoz's JSON error response.
  3. Trace-Log Context Ingestion: OpenTelemetry Winston instrumentation requires standard stdout JSON formatting to reliably extract trace_id and span_id.

Conclusion & Links

Building this pack demonstrated how powerful OpenTelemetry and SigNoz are when all three signals work in unison. Instead of staring at disconnected graphs during an incident, engineering teams get a single pane of glass showing budget remaining, exact firing alerts, and trace-correlated logs within seconds.

Top comments (0)