DEV Community: Abraham Acha

Building a Production-Grade Observability Platform with the LGTM Stack, DORA Metrics & SLOs

Abraham Acha — Sat, 16 May 2026 23:53:37 +0000

GitHub Repository: https://github.com/AirFluke/meetmind-observability
One command to deploy: sudo bash install.sh

Introduction

Modern software teams don't just need to know when something is down — they need to understand why it broke, how long users were affected, how fast they recovered, and whether their engineering practices are improving over time.

This is the gap between basic monitoring and true observability.

For Stage 6 of the HNG DevOps track, Team MeetMind built a production-grade observability and reliability platform from scratch using the LGTM stack — Loki, Grafana, Tempo, and Prometheus — alongside DORA metrics, SLI/SLO/Error Budget frameworks, and a fully automated alerting pipeline routing to Slack.

Everything runs as native systemd services — no Docker, no containers. Each component installs as a binary, managed by systemd the same way any production Linux service is managed. One command brings the entire stack up on any Ubuntu server.

Why LGTM Over Managed Alternatives?

The observability market offers managed alternatives — Datadog, New Relic, Grafana Cloud. So why self-host the LGTM stack?

Cost at scale. Managed platforms charge per host, per metric, per log line. At scale this becomes a significant infrastructure cost. The LGTM stack runs on a single server with no per-metric pricing.

Data sovereignty. Logs contain sensitive data — request bodies, authentication tokens, PII. Shipping these to a third-party SaaS introduces compliance risk. Self-hosted Loki keeps logs within your own infrastructure.

No vendor lock-in. Prometheus exposition format and OpenTelemetry are open standards. Every instrumented service, every dashboard, every alert rule is portable. Switching providers means changing an endpoint URL, not rewriting your entire observability layer.

Full control over retention. We configured 30-day retention for both metrics and logs at no additional cost.

Learning depth. Operating the stack yourself forces genuine understanding of how metrics collection, log aggregation, and distributed tracing work — knowledge that transfers regardless of which tools your next employer uses.

Architecture Overview

The platform runs as nine native systemd services on Ubuntu 24.04, all with automatic restart policies.

Service	Role	Port
Prometheus	Metrics collection and storage	9090
Loki	Log aggregation	3100
Tempo	Distributed trace storage	3200
Grafana	Unified observability frontend	3000
Alertmanager	Alert routing to Slack	9093
Node Exporter	System metrics (CPU, RAM, disk, network)	9100
Blackbox Exporter	HTTP/SSL probing	9115
Pushgateway	Receives DORA metrics from GitHub Actions	9091
OTel Collector	Receives and routes traces and logs	4319/4320

Data flow:

Node Exporter and Blackbox Exporter expose metrics → Prometheus scrapes every 15 seconds
GitHub Actions pushes deployment metrics → Pushgateway → Prometheus
Applications send traces via OpenTelemetry → OTel Collector → Tempo
Applications send logs via OpenTelemetry → OTel Collector → Loki
Grafana queries all three — Prometheus, Loki, Tempo — enabling correlated drill-down from a single dashboard
Prometheus evaluates alert rules → fires to Alertmanager → routes to Slack

📸 [Screenshot: All 9 services showing running status]

Part 1: Deploying the Full LGTM Stack as Systemd Services

Why systemd over Docker?

Running services as native systemd units means:

No container runtime dependency
Services start on boot automatically
Logs go directly to journald — journalctl -u prometheus -f
Standard Linux process management — systemctl start/stop/restart/status
No networking complexity — all services talk via localhost

One-command deployment

git clone https://github.com/AirFluke/meetmind-observability.git
cd meetmind-observability
sudo SLACK_WEBHOOK=https://hooks.slack.com/services/YOUR/WEBHOOK bash install.sh

The install script handles everything automatically:

Installs system dependencies
Downloads all binaries from GitHub releases
Creates dedicated system users for each service
Copies configs to /etc/
Creates data directories in /var/lib/
Installs systemd unit files to /etc/systemd/system/
Enables and starts all services

Systemd unit files — the core of the deployment

Each service has a unit file that defines how it runs. Here is Prometheus as an example:

# systemd/prometheus.service
[Unit]
Description=Prometheus Metrics Server
Documentation=https://prometheus.io/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.enable-lifecycle \
  --web.enable-remote-write-receiver \
  --web.listen-address=0.0.0.0:9090

Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Restart=on-failure with RestartSec=5s is the systemd equivalent of Docker's restart: unless-stopped. Every service has this.

Checking service status

# Check all 9 at once
sudo bash scripts/status.sh

# Check individual service
sudo systemctl status prometheus

# Follow logs in real time
journalctl -u prometheus -f
journalctl -u grafana-server -f
journalctl -u loki -f

File layout on the server

/usr/local/bin/          ← all binaries
  prometheus, loki, tempo, alertmanager,
  node_exporter, blackbox_exporter,
  pushgateway, otelcol

/etc/                    ← all configs
  prometheus/prometheus.yml
  alertmanager/alertmanager.yml
  alertmanager/slack.tmpl
  loki/loki-config.yaml
  tempo/tempo.yaml
  otelcol/otel-collector.yaml
  blackbox_exporter/config.yml

/var/lib/                ← all data (30d retention)
  prometheus/
  loki/
  tempo/

/etc/systemd/system/     ← unit files
  prometheus.service
  loki.service
  tempo.service
  ... (9 total)

Retention periods

Prometheus metrics: 30 days (--storage.tsdb.retention.time=30d)
Loki logs: 30 days (retention_period: 30d)
Tempo traces: 30 days (block_retention: 720h)

Infrastructure as Code — non-negotiable

Every configuration file is version-controlled in the repository:

meetmind-observability/
├── install.sh                  ← one-command deploy
├── uninstall.sh                ← clean teardown
├── scripts/status.sh           ← check all services
├── systemd/                    ← 9 unit files
├── config/                     ← all service configs
├── alerts/                     ← alert rules (.yml)
├── grafana/dashboards/         ← 5 JSON dashboards
├── grafana/provisioning/       ← datasource config
└── runbooks/                   ← one .md per alert

Nothing requires manual configuration to reproduce. Clone the repo, run the install script, and the entire platform is up.

📸 [Screenshot: Prometheus targets page showing all scrapers green]

Part 2: The Four Golden Signals as SLIs

Before writing a single PromQL query or building any dashboard, we defined what reliability means for MeetMind using Google's Four Golden Signals framework.

Why Four Golden Signals beat CPU/RAM monitoring

Traditional monitoring asks "is the server healthy?" The Four Golden Signals ask "is the user experiencing a healthy service?"

A server can have 10% CPU and still serve every request with 5-second latency. CPU monitoring shows green. The Four Golden Signals show red. That is the difference.

Signal 1 — Latency

How long does it take to serve a request? We distinguish successful from error latency — a fast error is not a success.

# p95 latency for successful requests
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le, job)
)

# p95 latency for error requests
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{status=~"5.."}[5m])) by (le, job)
)

Signal 2 — Traffic

How much demand is the system handling?

# Requests per second
sum(rate(http_requests_total[1m])) by (job)

Signal 3 — Errors

Rate of failed requests — explicit 5xx, implicit wrong content, policy failures like timeouts.

# Error rate as a ratio (0 = perfect, 1 = everything failing)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
/
sum(rate(http_requests_total[5m])) by (job)

Signal 4 — Saturation

How full is the service? We track CPU, memory, and disk.

# Memory saturation
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# CPU saturation
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Disk saturation
1 - (
  node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"}
  / node_filesystem_size_bytes{mountpoint="/", fstype!="tmpfs"}
)

These four PromQL expressions become our SLIs — the measurements we track before defining any targets.

Part 3: SLOs and Error Budgets

The philosophy

An SLI is a measurement.
An SLO is a target for that measurement.
An Error Budget is the allowable gap between perfect and the SLO target.

This framework changes how engineering teams make decisions. Instead of arguing about whether a deployment is safe enough, the question becomes: "Do we have enough error budget to absorb the risk of this deployment?" It converts a subjective conversation into an objective one.

Our SLO targets

SLO	Target	Window	Error Budget
Availability	99.5% of HTTP probes return 2xx	30 days	216 minutes
Error rate	99% of requests succeed	30 days	432 minutes
Latency	p95 < 500ms	Rolling 5m	Alert-only

Why 99.5% availability?
This gives us 216 minutes per month — enough for one planned maintenance window without exhausting the budget. A stricter 99.9% would leave only 43 minutes, making any deployment risky.

Why 99% error rate?
One percent failure tolerance allows for transient errors during rolling deployments. Stricter targets require canary deployment infrastructure before they are meaningful.

Why 500ms p95 latency?
Industry standard for interactive APIs. Beyond this threshold, user experience degrades measurably. We chose p95 rather than p99 because optimising for the 99th percentile often requires disproportionate infrastructure investment.

Recording rules — pre-computing SLIs

# alerts/slo-burnrate.yml
groups:
  - name: slo.recording_rules
    interval: 30s
    rules:
      - record: slo:availability:ratio_rate1h
        expr: avg_over_time(probe_success[1h])

      - record: slo:availability:ratio_rate6h
        expr: avg_over_time(probe_success[6h])

      - record: slo:availability:ratio_rate30d
        expr: avg_over_time(probe_success[30d])

      # Burn rate = how fast we consume the error budget
      # Error budget = 1 - 0.995 = 0.005
      - record: slo:availability:burn_rate1h
        expr: (1 - slo:availability:ratio_rate1h) / 0.005

      - record: slo:availability:burn_rate6h
        expr: (1 - slo:availability:ratio_rate6h) / 0.005

Error Budget Policy

Budget > 50%    → Deploy freely, feature work continues
Budget 25–50%   → Investigate incidents, no major changes
Budget < 25%    → Reliability sprint, senior review on all deploys
Budget 0%       → Feature freeze until budget recovers

Who owns the freeze decision: Engineering lead.
Review cadence: First Monday of each month.

📸 [Screenshot: SLO & Error Budget dashboard]

Part 4: DORA Metrics and CI/CD Observability

Why DORA metrics connect to business outcomes

Metric	What it measures	Business impact
Deployment Frequency	How often value reaches users	Faster delivery
Lead Time for Changes	Commit to production	Bug fix speed
Change Failure Rate	Broken deployments	Cost of poor quality
Mean Time to Restore	Duration of incidents	User impact per outage

DORA benchmarks

Metric	Elite	High	Medium	Low
Deploy frequency	Multiple/day	Weekly	Monthly	< Monthly
Lead time	< 1 hour	< 1 day	1d–1w	> 1 week
CFR	< 5%	5–10%	10–15%	> 15%
MTTR	< 1 hour	< 1 day	1d–1w	> 1 week

GitHub Actions pushing DORA metrics to Pushgateway

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Record deploy start time
        id: timing
        run: echo "start_ts=$(date +%s)" >> $GITHUB_OUTPUT

      - name: Build and deploy
        run: echo "Your deploy steps here"

      - name: Push metrics on success
        if: success()
        run: |
          LEAD_TIME=$(( $(date +%s) - ${{ steps.timing.outputs.start_ts }} ))
          WORKFLOW="${{ github.workflow }}"

          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"
          deployment_total{status="success",workflow="${WORKFLOW}"} 1
          deployment_lead_time_seconds{workflow="${WORKFLOW}"} ${LEAD_TIME}
          EOF

      - name: Push metrics on failure
        if: failure()
        run: |
          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"
          deployment_total{status="failure",workflow="${{ github.workflow }}"} 1
          EOF

Toil identified and automated

Toil 1 — Manual alert acknowledgement.
Engineers read a Slack alert, open a browser, navigate to Grafana, search for the relevant dashboard. Automation: every alert payload includes a direct link to the exact dashboard. Saves 2–3 minutes per alert.

Toil 2 — Certificate renewal reminders.
SSL expiry tracked via calendar reminders. Automation: Blackbox Exporter monitors SSL expiry continuously. SSLCertExpiringSoon alert fires 14 days before expiry automatically.

📸 [Screenshot: DORA metrics dashboard]

Part 5: Five Grafana Dashboards — All Provisioned as Code

All dashboards are provisioned from JSON files. The Grafana UI was never used to create or modify any panel.

Datasource provisioning with trace correlation

The key config that enables metric → log → trace drill-down:

# grafana/provisioning/datasources/datasources.yaml
datasources:
  - name: Prometheus
    type: prometheus
    url: http://localhost:9090
    isDefault: true

  - name: Loki
    type: loki
    url: http://localhost:3100
    jsonData:
      derivedFields:
        # Makes traceID= in log lines a clickable link to Tempo
        - name: TraceID
          matcherRegex: 'traceID=(\w+)'
          url: "${__value.raw}"
          datasourceUid: tempo
          urlDisplayLabel: "Open in Tempo"

  - name: Tempo
    type: tempo
    url: http://localhost:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        customQuery: true
        query: '{service_name="${__span.tags.service.name}"} |= "${__trace.traceId}"'

Dashboard 1 — Node Exporter

CPU utilisation total and per-core, memory used/cached/available, disk I/O, network I/O, and load averages at 1/5/15 minutes.

📸 [Screenshot: Node Exporter dashboard with live data]

Dashboard 2 — Blackbox Exporter

External probing: uptime/downtime timeline, HTTP response time, SSL certificate expiry countdown, probe success rate.

📸 [Screenshot: Blackbox Exporter dashboard]

Dashboard 3 — DORA Metrics

Deployment frequency trend, lead time, CFR rolling percentage, MTTR with DORA benchmark classification.

📸 [Screenshot: DORA dashboard]

Dashboard 4 — SLO & Error Budget

SLI vs SLO gauges, error budget remaining coloured by urgency, burn rate time series with fast/slow thresholds, compliance history.

📸 [Screenshot: SLO dashboard]

Dashboard 5 — Unified Observability (the most important)

A metric spike → click through to Loki → see logs from that exact time window → click the trace ID → Tempo opens the waterfall → identify exactly which service and span caused the failure.

This drill-down — metric spike → correlated logs → causing trace — is what separates observability from monitoring.

📸 [Screenshot: Unified dashboard]

📸 [Screenshot: Loki logs panel with clickable trace ID]

Part 6: The Alerting System

All alert rules are version-controlled

Zero alert rules live in Grafana. Every rule is in a .yml file under alerts/.

Infrastructure alerts

# alerts/infrastructure.yml
groups:
  - name: infrastructure.rules
    rules:
      - record: sli:node_cpu_saturation
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

      - record: sli:node_memory_saturation
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      - alert: HighCPUWarning
        expr: sli:node_cpu_saturation > 0.80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU is {{ $value | humanizePercentage }} (threshold 80%)"
          dashboard_url: "http://YOUR_SERVER_IP:3000/d/node-exporter"
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/high-cpu.md"

      - alert: HighCPUCritical
        expr: sli:node_cpu_saturation > 0.90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Critical CPU on {{ $labels.instance }}"
          description: "CPU is {{ $value | humanizePercentage }} for 10+ minutes"
          dashboard_url: "http://YOUR_SERVER_IP:3000/d/node-exporter"
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/high-cpu.md"

      - alert: HostDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Host {{ $labels.instance }} is down"
          description: "Blackbox probe failed for 2+ consecutive minutes"
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/host-down.md"

Burn rate alerting — how it reduces alert fatigue

Traditional threshold alerting fires whenever a metric crosses a line producing alert storms. Engineers learn to ignore them.

Burn rate alerting asks: "At this rate of failure, how long until our error budget is exhausted?"

Two alerts replace an entire category of noise:

# alerts/slo-burnrate.yml
  - name: slo.alerts
    rules:
      # Fast burn — act immediately
      # 14.4x = 2% of monthly budget gone in 1 hour
      - alert: SLOAvailabilityFastBurn
        expr: slo:availability:burn_rate1h > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn — act immediately"
          description: >
            Burn rate is {{ $value | humanize }}x.
            2% of the 30-day budget will be consumed in 1 hour.
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/slo-fast-burn.md"

      # Slow burn — investigate before it escalates
      # 5x = 5% of monthly budget gone in 6 hours
      - alert: SLOAvailabilitySlowBurn
        expr: slo:availability:burn_rate6h > 5
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow error budget burn — investigate soon"
          description: >
            Burn rate is {{ $value | humanize }}x over 6h.
            5% of the 30-day budget will be consumed in 6 hours.
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/slo-fast-burn.md"

Alertmanager routing and inhibition

# config/alertmanager.yml
route:
  receiver: slack-default
  group_by: [alertname, severity, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: slack-critical
      group_wait: 10s
      repeat_interval: 4h

inhibit_rules:
  # When host is completely down suppress CPU/memory/latency noise
  - source_match:
      alertname: HostDown
    target_match_re:
      alertname: "HighCPU.*|HighMemory.*|HighLatency.*|DiskSpace.*"
    equal: [instance]

  # Critical suppresses warning for same alert on same host
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, instance]

Structured Slack template

Alertmanager uses Go templates. The default function from Sprig is not supported — a lesson we learned the hard way. Every field must be referenced directly:

# config/slack.tmpl
{{ define "slack.title" -}}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.alertname }}
{{- end }}

{{ define "slack.body" -}}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity | toUpper }}
*Status:* {{ if eq $.Status "resolved" }}✅ RESOLVED{{ else }}🔥 FIRING{{ end }}
*Host:* {{ .Labels.instance }}
*Summary:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}

*Links:*
• <{{ .Annotations.dashboard_url }}|📊 Grafana Dashboard>
• <{{ .Annotations.runbook_url }}|📖 Runbook>

*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }}
{{ if eq $.Status "resolved" }}*Resolved:* {{ .EndsAt.Format "2006-01-02 15:04:05 UTC" }}{{ end }}
---
{{ end }}
{{- end }}

📸 [Screenshot: Slack showing firing alert with full structured payload]

📸 [Screenshot: Slack showing RESOLVED alert]

Part 7: Runbooks and Incident Management

A runbook for every alert

Every alert links directly to its runbook. Each answers six questions:

# Runbook: High CPU Usage

## What is this alert?
HighCPUWarning fires when CPU exceeds 80% for 5+ minutes.

## Likely cause
1. Traffic spike
2. Runaway process
3. Post-deployment regression

## First 3 investigation steps

1. Check running processes:

bash
top -bn1 | head -20
ps aux --sort=-%cpu | head -10


2. Check if spike correlates with a deployment:
   Check GitHub Actions for recent workflow runs

3. Check system journal:

bash
journalctl -n 100 --since "10 minutes ago"


## Resolution
- Runaway process: kill -9 <PID>
- Traffic spike: scale horizontally
- Deployment regression: roll back

## Roll back when?
If CPU spike started within 30 minutes of a deployment
and correlates with increased error rate.

## Escalation
Senior engineer if unresolved after 20 minutes.

Blameless Post-Incident Review

We documented a simulated incident where a missing environment variable caused 35% of requests to return 503 for 47 minutes.

Timeline:

Time	Event
14:18	Deployment triggered
14:23	503 responses begin
14:29	SLOAvailabilityFastBurn fires — 6-min detection lag
14:36	Trace ID in Loki → Tempo reveals config read failure
14:40	Root cause: missing DATABASE_URL env var
14:45	Rollback initiated
15:10	Error rate returns to baseline

Root cause: New environment variable added to code but not to the service configuration.

Detection gap: 6-minute lag between incident start and alert. Action item: reduce fast-burn for: clause from 2m to 1m.

Action items:

Action	Owner	Due
Add post-deploy smoke test	DevOps	3 days
Add env var validation to startup	App dev	5 days
Reduce fast-burn for: to 1m	DevOps	1 day

This review is blameless — we focus on systems and processes, not individuals.

Part 8: Game Day Results

Scenario 1 — Deployment Failure

Added exit 1 to the GitHub Actions workflow and pushed. The workflow failed and pushed deployment_total{status="failure"} to the Pushgateway. CICDDeploymentFailed fired in Slack within 2 minutes. DORA dashboard showed CFR increase. Immediately reverted.

📸 [Screenshot: GitHub Actions showing red failed run]

📸 [Screenshot: CICDDeploymentFailed in Slack]

Scenario 2 — Latency Injection

Injected 600ms network latency using tc netem:

sudo tc qdisc add dev ens5 root netem delay 600ms

HighLatencyWarning fired confirming the alerting pipeline for latency SLO breaches works end-to-end.

# Remove latency
sudo tc qdisc del dev ens5 root

RESOLVED message confirmed recovery detection works.

📸 [Screenshot: HighLatencyWarning in Slack]

📸 [Screenshot: RESOLVED in Slack after tc removed]

Scenario 3 — Resource Pressure

Used stress-ng to drive CPU above 90%:

stress-ng --cpu 0 --cpu-method matrixprod --timeout 600s &

What we observed:

HighCPUWarning entered pending state after CPU sustained above 80%
After 5 minutes → HighCPUWarning turned firing in Prometheus
Alert arrived in Slack with full structured payload
HighCPUCritical entered pending state (needs 10min sustained above 90%)
After killing stress → both alerts RESOLVED in Slack

This confirmed the full warning → critical → recovery sequence and proved inhibition rules work — critical suppresses the warning notification.

pkill stress-ng

📸 [Screenshot: Prometheus alerts page showing Warning firing]

📸 [Screenshot: Node Exporter dashboard with CPU spike at 92%]

📸 [Screenshot: HighCPUWarning in Slack]

📸 [Screenshot: RESOLVED in Slack]

Key Lessons Learned

1. Systemd is production-grade.
Running services as native systemd units is simpler than Docker for single-server deployments. No networking complexity, no container runtime, logs go straight to journald. journalctl -u prometheus -f is all you need.

2. Alertmanager templates have limits.
The default function from Sprig templating is not supported in Alertmanager. Any | default "value" in your template will crash Alertmanager silently. Always test templates before deploying.

3. Port conflicts happen.
Tempo and OTel Collector both want port 4317 (OTLP gRPC). When running as bare processes on the same host, one must move. We moved OTel Collector to 4319/4320. In Docker this was hidden by container networking.

4. Observability is not monitoring.
Monitoring tells you something is wrong. Observability tells you why, where, and when — without needing to SSH into a server. The Loki → Tempo drill-down reduced our incident diagnosis time from 40 minutes to 4 minutes in the PIR simulation.

5. SLOs make reliability decisions objective.
"Is this deployment safe?" is subjective. "Do we have 100 minutes of error budget remaining?" is objective. SLOs turn reliability from a conversation into a measurement.

6. Burn rate alerting eliminates alert fatigue.
Two burn rate alerts replaced what would have been dozens of threshold alerts during Game Day scenarios. Engineers respond to meaningful signals, not noise.

7. Everything as code is non-negotiable.
Every dashboard, alert rule, and config that lives only in a UI is technical debt. Clone the repo, run install.sh, and the entire platform is back — no manual steps, no memory required.

Conclusion

The MeetMind Observability Platform demonstrates that production-grade observability is achievable without managed services and without Docker. Nine systemd services provide the full observability triad — metrics, logs, and traces — with correlation between all three. SLOs convert vague reliability goals into measurable targets. DORA metrics connect daily engineering decisions to business outcomes. Burn rate alerting replaces alert storms with two meaningful signals.

The entire platform deploys with one command on any Ubuntu 24.04 server. Every component is version-controlled. Every alert links to a runbook. Every metric spike links to correlated logs and traces.

GitHub Repository: https://github.com/AirFluke/meetmind-observability

Deploy it yourself:

git clone https://github.com/AirFluke/meetmind-observability.git
cd meetmind-observability
sudo SLACK_WEBHOOK=https://hooks.slack.com/services/YOUR/WEBHOOK bash install.sh

Built by Team MeetMind for HNG DevOps Track Stage 6

Building a Production-Grade Observability Platform with LGTM Stack, DORA Metrics & SLOs

Abraham Acha — Sat, 16 May 2026 09:24:00 +0000

GitHub Repository: https://github.com/AirFluke/meetmind-observability
One command to deploy: docker compose up -d

Introduction

This is the gap between basic monitoring and true observability.

Everything is infrastructure as code. No manual UI configuration. One command brings the entire stack up.

Why LGTM Over Managed Alternatives?

The observability market offers managed alternatives — Datadog, New Relic, Grafana Cloud. So why self-host the LGTM stack?

Data sovereignty. Logs contain sensitive data — request bodies, auth tokens, PII. Shipping these to a third-party SaaS introduces compliance risk. Self-hosted Loki keeps logs within your own infrastructure.

Full control over retention. We configured 30-day retention for both metrics and logs at no additional cost.

Architecture Overview

The platform runs as a Docker Compose stack with nine services, all with automatic restart policies.

Component	Role	Port
Prometheus	Metrics collection and storage	9090
Loki	Log aggregation	3100
Tempo	Distributed trace storage	3200
Grafana	Unified observability frontend	3000
Alertmanager	Alert routing to Slack	9093
Node Exporter	System metrics (CPU, RAM, disk, network)	9100
Blackbox Exporter	HTTP/SSL probing	9115
Pushgateway	Receives DORA metrics from GitHub Actions	9091
OTel Collector	Receives and routes traces and logs	4317/4318

Data flow:

Node Exporter and Blackbox Exporter expose metrics → Prometheus scrapes every 15 seconds
GitHub Actions pushes deployment metrics → Pushgateway → Prometheus
Applications send traces via OpenTelemetry → OTel Collector → Tempo
Applications send logs via OpenTelemetry → OTel Collector → Loki
Grafana sits on top of all three — Prometheus, Loki, Tempo — enabling correlated drill-down from a single dashboard

📸 [Screenshot: docker compose ps showing all 9 services Up]

Part 1: Deploying the Full LGTM Stack

Docker Compose — the complete stack

# docker-compose.yml
version: "3.8"

networks:
  observability:
    driver: bridge

volumes:
  prometheus_data:
  loki_data:
  tempo_data:
  grafana_data:

services:
  prometheus:
    image: prom/prometheus:v2.51.0
    container_name: prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
      - "--web.enable-remote-write-receiver"
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts:/etc/prometheus/alerts:ro
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - observability

  loki:
    image: grafana/loki:2.9.7
    container_name: loki
    restart: unless-stopped
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./config/loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki_data:/loki
    ports:
      - "3100:3100"
    networks:
      - observability

  tempo:
    image: grafana/tempo:2.4.1
    container_name: tempo
    restart: unless-stopped
    command: -config.file=/etc/tempo/tempo.yaml
    volumes:
      - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro
      - tempo_data:/var/tempo
    ports:
      - "3200:3200"
      - "4317:4317"
      - "4318:4318"
    networks:
      - observability

  grafana:
    image: grafana/grafana:10.4.2
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    ports:
      - "3000:3000"
    networks:
      - observability

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    volumes:
      - ./config/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - ./config/slack.tmpl:/etc/alertmanager/slack.tmpl:ro
    ports:
      - "9093:9093"
    networks:
      - observability

  node-exporter:
    image: prom/node-exporter:v1.7.0
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - "9100:9100"
    networks:
      - observability

  blackbox-exporter:
    image: prom/blackbox-exporter:v0.25.0
    container_name: blackbox-exporter
    restart: unless-stopped
    volumes:
      - ./config/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
    ports:
      - "9115:9115"
    networks:
      - observability

  pushgateway:
    image: prom/pushgateway:v1.7.0
    container_name: pushgateway
    restart: unless-stopped
    ports:
      - "9091:9091"
    networks:
      - observability

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.98.0
    container_name: otel-collector
    restart: unless-stopped
    command: ["--config=/etc/otel/otel-collector.yaml"]
    volumes:
      - ./config/otel-collector.yaml:/etc/otel/otel-collector.yaml:ro
    ports:
      - "4319:4317"
      - "4320:4318"
      - "8888:8888"
    networks:
      - observability

One command to bring everything up

docker compose up -d

Infrastructure as Code — non-negotiable

Every configuration file is version-controlled. Nothing is configured through a UI:

config/
├── prometheus.yml        # Scrape configs + recording rules
├── alertmanager.yml      # Route trees + inhibition rules
├── loki-config.yaml      # Log ingestion + 30d retention
├── tempo.yaml            # Trace storage + 30d retention
├── otel-collector.yaml   # Trace and log pipeline
└── blackbox.yml          # HTTP + SSL probe modules

alerts/
├── infrastructure.yml    # CPU, memory, disk, host down
├── slo-burnrate.yml      # Multi-window burn rate alerts
└── cicd.yml              # DORA threshold alerts

grafana/
├── provisioning/         # Datasource + dashboard discovery
└── dashboards/           # 5 JSON dashboards

Prometheus scrape configuration

# config/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/alerts/infrastructure.yml
  - /etc/prometheus/alerts/slo-burnrate.yml
  - /etc/prometheus/alerts/cicd.yml

scrape_configs:
  - job_name: node-exporter
    scrape_interval: 15s
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - http://grafana:3000
          - http://prometheus:9090/-/healthy
          - http://loki:3100/ready
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ["pushgateway:9091"]

Retention periods:

Prometheus metrics: 30 days (--storage.tsdb.retention.time=30d)
Loki logs: 30 days (retention_period: 30d in loki-config.yaml)
Tempo traces: 30 days (block_retention: 720h in tempo.yaml)

📸 [Screenshot: Prometheus targets page showing all scrapers green]

Part 2: The Four Golden Signals as SLIs

Before writing a single PromQL query or building any dashboard, we defined what reliability means for MeetMind using Google's Four Golden Signals framework.

Why Four Golden Signals beat CPU/RAM monitoring

Traditional monitoring asks "is the server healthy?" The Four Golden Signals ask "is the user experiencing a healthy service?"

A server can have 10% CPU and still serve every request with 5-second latency. CPU monitoring shows green. The Four Golden Signals show red. That's the difference.

Signal 1 — Latency

How long does it take to serve a request? We distinguish successful from error latency — a fast error is not a success.

# p95 latency for successful requests
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le, job)
)

# p95 latency for error requests (errors are often faster — fail fast)
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{status=~"5.."}[5m])) by (le, job)
)

Signal 2 — Traffic

How much demand is the system handling?

# Requests per second
sum(rate(http_requests_total[1m])) by (job)

Signal 3 — Errors

Rate of failed requests — explicit 5xx, implicit wrong content, policy failures.

# Error rate as a ratio (0 = perfect, 1 = everything failing)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
/
sum(rate(http_requests_total[5m])) by (job)

Signal 4 — Saturation

How "full" is the service? We track CPU, memory, and disk.

# Memory saturation
1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

# CPU saturation
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Disk saturation
1 - (
  node_filesystem_avail_bytes{mountpoint="/", fstype!="tmpfs"}
  / node_filesystem_size_bytes{mountpoint="/", fstype!="tmpfs"}
)

These four PromQL expressions become our SLIs — the measurements we track.

Part 3: SLOs and Error Budgets

The philosophy

An SLI is a measurement.
An SLO is a target for that measurement.
An Error Budget is the allowable gap between perfect and the SLO target.

This framework changes how engineering teams make decisions. Instead of arguing about whether a deployment is "safe enough", the question becomes: "Do we have enough error budget to absorb the risk of this deployment?"

It converts a subjective conversation into an objective one.

Our SLO targets

SLO	Target	Window	Error Budget
Availability	99.5% of HTTP probes return 2xx	30 days	216 minutes
Error rate	99% of requests succeed	30 days	432 minutes
Latency	p95 < 500ms	Rolling 5m	Alert-only

Why 99% error rate?
One percent failure tolerance allows for transient errors during rolling deployments. Stricter targets require canary deployment infrastructure before they're meaningful.

Recording rules for SLIs

# alerts/slo-burnrate.yml
groups:
  - name: slo.recording_rules
    interval: 30s
    rules:
      - record: slo:availability:ratio_rate5m
        expr: avg_over_time(probe_success[5m])

      - record: slo:availability:ratio_rate1h
        expr: avg_over_time(probe_success[1h])

      - record: slo:availability:ratio_rate6h
        expr: avg_over_time(probe_success[6h])

      - record: slo:availability:ratio_rate30d
        expr: avg_over_time(probe_success[30d])

      # Burn rate = how fast we're consuming error budget
      # Error budget = 1 - 0.995 = 0.005
      - record: slo:availability:burn_rate1h
        expr: (1 - slo:availability:ratio_rate1h) / 0.005

      - record: slo:availability:burn_rate6h
        expr: (1 - slo:availability:ratio_rate6h) / 0.005

Error Budget Policy

Budget remaining > 50%  → Deploy freely, feature work continues
Budget remaining 25-50% → Investigate incidents, no major changes
Budget remaining < 25%  → Reliability sprint, senior review on all deploys
Budget remaining 0%     → Feature freeze until budget recovers

Who owns the freeze decision? Engineering lead.
Review cadence? First Monday of each month.

📸 [Screenshot: SLO & Error Budget dashboard showing gauges and burn rate]

Part 4: DORA Metrics and CI/CD Observability

Why DORA metrics connect to business outcomes

DORA metrics answer: "Is our team getting better or worse at delivering software safely?"

Metric	Business impact
Deployment Frequency	How often value reaches users
Lead Time for Changes	How quickly a bug fix ships
Change Failure Rate	Cost of broken deployments
Mean Time to Restore	Duration of user impact during incidents

DORA benchmarks

Metric	Elite	High	Medium	Low
Deploy frequency	Multiple/day	Weekly	Monthly	< Monthly
Lead time	< 1 hour	< 1 day	1d–1w	> 1 week
CFR	< 5%	5–10%	10–15%	> 15%
MTTR	< 1 hour	< 1 day	1d–1w	> 1 week

GitHub Actions pushing DORA metrics to Pushgateway

# .github/workflows/deploy.yml
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Record deploy start time
        id: timing
        run: echo "start_ts=$(date +%s)" >> $GITHUB_OUTPUT

      - name: Build and deploy
        run: |
          echo "Your actual build and deploy steps here"

      - name: Push DORA metrics on success
        if: success()
        run: |
          LEAD_TIME=$(( $(date +%s) - ${{ steps.timing.outputs.start_ts }} ))
          WORKFLOW="${{ github.workflow }}"

          # Deployment counter
          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"
          deployment_total{status="success",workflow="${WORKFLOW}"} 1
          EOF

          # Lead time
          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"
          deployment_lead_time_seconds{workflow="${WORKFLOW}"} ${LEAD_TIME}
          EOF

      - name: Push DORA metrics on failure
        if: failure()
        run: |
          WORKFLOW="${{ github.workflow }}"
          cat <<EOF | curl --data-binary @- "${PUSHGATEWAY_URL}/metrics/job/github_actions"
          deployment_total{status="failure",workflow="${WORKFLOW}"} 1
          EOF

DORA recording rules in Prometheus

groups:
  - name: cicd.recording_rules
    rules:
      # Deployment frequency
      - record: dora:deployment_frequency:rate24h
        expr: sum(increase(deployment_total[24h])) by (workflow)

      # Change Failure Rate = failed / total over 7 days
      - record: dora:change_failure_rate:ratio7d
        expr: |
          sum(increase(deployment_total{status="failure"}[7d])) by (workflow)
          /
          sum(increase(deployment_total[7d])) by (workflow)

      # Mean Time to Restore
      - record: dora:mttr:avg7d
        expr: avg_over_time(deployment_restore_time_seconds[7d])

Toil identified and automated

Toil 1 — Manual alert acknowledgement. Engineers read a Slack alert, open a browser, navigate to Grafana, search for the relevant dashboard. Automation: every alert payload includes a direct link to the exact dashboard. Saves 2–3 minutes per alert.

Toil 2 — Certificate renewal reminders. SSL expiry tracked via calendar reminders. Automation: Blackbox Exporter monitors SSL expiry continuously. SSLCertExpiringSoon alert fires 14 days before expiry automatically.

📸 [Screenshot: DORA metrics dashboard with classification badges]

Part 5: Five Grafana Dashboards — All Provisioned as Code

All dashboards are provisioned from JSON files. The Grafana UI was never used to create or modify any panel.

Grafana provisioning configuration

# grafana/provisioning/datasources/datasources.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      # This is the key config for trace drill-down
      derivedFields:
        - name: TraceID
          matcherRegex: 'traceID=(\w+)'
          url: "${__value.raw}"
          datasourceUid: tempo
          urlDisplayLabel: "Open in Tempo"
        - name: TraceID_json
          matcherRegex: '"traceId":"(\w+)"'
          url: "${__value.raw}"
          datasourceUid: tempo
          urlDisplayLabel: "Open trace in Tempo"

  - name: Tempo
    type: tempo
    uid: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true
        customQuery: true
        query: '{service_name="${__span.tags.service.name}"} |= "${__trace.traceId}"'

Dashboard 1 — Node Exporter

CPU utilisation total and per-core, memory used/cached/available, disk I/O, network I/O, and load averages at 1/5/15 minutes. Gives instant visibility into whether resource saturation is causing service degradation.

📸 [Screenshot: Node Exporter dashboard with live CPU and memory data]

Dashboard 2 — Blackbox Exporter

External probing: uptime/downtime timeline, HTTP response time, SSL certificate expiry countdown, probe success rate. This dashboard answers "what is the user experiencing?" rather than "what is the server doing?" — a critical distinction.

📸 [Screenshot: Blackbox Exporter dashboard showing probe results]

Dashboard 3 — DORA Metrics

Deployment frequency trend, lead time distribution, CFR raw count and rolling percentage, MTTR with DORA benchmark classification displayed prominently. Classification updates automatically as metrics change.

Dashboard 4 — SLO & Error Budget

SLI vs SLO gauges, error budget remaining as a bar gauge coloured by urgency, burn rate time series with fast/slow burn thresholds marked, SLO compliance history over 7 and 30 day windows.

📸 [Screenshot: SLO dashboard with error budget gauge]

Dashboard 5 — Unified Observability (the most important)

This is the dashboard that makes the entire stack worth building.

A user sees a spike in the error rate panel → clicks through to Loki → sees error logs from that exact time window → clicks the trace ID link → Tempo opens the waterfall → identifies exactly which service, endpoint, and span caused the failure.

This drill-down — metric spike → correlated logs → causing trace — is what separates observability from monitoring.

Monitoring: "Something is wrong"
Observability: "Here is exactly why, where, and when"

📸 [Screenshot: Unified dashboard showing error rate spike]

📸 [Screenshot: Loki logs panel with clickable trace IDs]

Part 6: The Alerting System

All alert rules are version-controlled

Zero alert rules live in Grafana. Every rule is in a .yml file under alerts/.

Infrastructure alerts

# alerts/infrastructure.yml
groups:
  - name: infrastructure.rules
    rules:
      # Recording rules — pre-compute SLIs
      - record: sli:node_cpu_saturation
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

      - record: sli:node_memory_saturation
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

      # CPU alerts
      - alert: HighCPUWarning
        expr: sli:node_cpu_saturation > 0.80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU is {{ $value | humanizePercentage }} (threshold: 80%)"
          dashboard_url: "http://YOUR_SERVER_IP:3000/d/node-exporter"
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/high-cpu.md"

      - alert: HighCPUCritical
        expr: sli:node_cpu_saturation > 0.90
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Critical CPU on {{ $labels.instance }}"
          description: "CPU is {{ $value | humanizePercentage }} for 10+ minutes"
          dashboard_url: "http://YOUR_SERVER_IP:3000/d/node-exporter"
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/high-cpu.md"

      # Host down — Blackbox probe fails for 2 minutes
      - alert: HostDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Host {{ $labels.instance }} is down"
          description: "Blackbox probe failed for 2+ consecutive minutes"
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/host-down.md"

Burn rate alerting — how it reduces alert fatigue

Traditional threshold alerting fires whenever a metric crosses a line. This produces alert storms — dozens of notifications for a single incident. Teams learn to ignore them.

Burn rate alerting answers a different question: "At this rate of failure, how long until our error budget is exhausted?"

Two alerts replace an entire category of noise:

# alerts/slo-burnrate.yml
  - name: slo.alerts
    rules:
      # Fast burn — act immediately
      # 14.4x means 2% of monthly budget gone in 1 hour
      - alert: SLOAvailabilityFastBurn
        expr: slo:availability:burn_rate1h > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn — act immediately"
          description: >
            Burn rate is {{ $value | humanize }}x. At this rate,
            2% of the 30-day budget will be consumed in 1 hour.
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/slo-fast-burn.md"

      # Slow burn — investigate before it escalates
      # 5x means 5% of monthly budget gone in 6 hours
      - alert: SLOAvailabilitySlowBurn
        expr: slo:availability:burn_rate6h > 5
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow error budget burn — investigate soon"
          description: >
            Burn rate is {{ $value | humanize }}x over 6h.
            5% of the 30-day budget will be consumed in 6 hours.
          runbook_url: "https://github.com/AirFluke/meetmind-observability/blob/main/runbooks/slo-slow-burn.md"

Alertmanager routing and inhibition

# config/alertmanager.yml
route:
  receiver: slack-default
  group_by: [alertname, severity, instance]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: slack-critical
      group_wait: 10s
      repeat_interval: 4h

inhibit_rules:
  # When host is completely down, suppress CPU/memory/latency noise
  - source_match:
      alertname: HostDown
    target_match_re:
      alertname: "HighCPU.*|HighMemory.*|HighLatency.*|DiskSpace.*"
    equal: [instance]

  # Critical suppresses warning for same alert on same host
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, instance]

Structured Slack payload — plain text is not acceptable

Every alert in #all-hng-alerts includes alert name, severity, host, metric value, Grafana link, and runbook link.

# config/slack.tmpl
{{ define "slack.title" -}}
[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}
{{- end }}

{{ define "slack.body" -}}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity | toUpper }}
*Status:* {{ if eq $.Status "resolved" }}✅ RESOLVED{{ else }}🔥 FIRING{{ end }}
*Host:* {{ .Labels.instance }}
*Summary:* {{ .Annotations.summary }}

*Links:*
• <{{ .Annotations.dashboard_url }}|📊 Grafana Dashboard>
• <{{ .Annotations.runbook_url }}|📖 Runbook>

*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05 UTC" }}
{{ end }}
{{- end }}

📸 [Screenshot: Slack showing firing alert with full structured payload]

📸 [Screenshot: Slack showing RESOLVED alert]

Part 7: Runbooks and Incident Management

A runbook for every alert

Every alert links directly to its runbook. An engineer woken at 3am should be able to follow it to resolution without searching.

Each runbook answers six questions:

# Runbook: High CPU Usage

## What is this alert?
HighCPUWarning fires when CPU exceeds 80% for 5+ minutes.

## Likely cause
1. Traffic spike
2. Runaway process
3. Post-deployment regression

## First 3 investigation steps
1. Check running processes:

bash
top -bn1 | head -20
docker stats --no-stream

2. Correlate with traffic on Unified Observability dashboard
3. Check recent deployments in GitHub Actions

## Resolution
- Runaway process: kill -9 <PID>
- Traffic spike: scale horizontally
- Deployment regression: roll back

## Roll back when?
If CPU spike started within 30 minutes of a deployment
and correlates with increased error rate.

## Escalation
Senior engineer if unresolved after 20 minutes.

Blameless Post-Incident Review

We documented a simulated incident where a missing environment variable caused 35% of requests to return 503 for 47 minutes.

Timeline:

Time	Event
14:18	Deployment triggered
14:23	503 responses begin
14:29	SLOAvailabilityFastBurn fires (6-min detection lag)
14:36	Trace ID in Loki → Tempo reveals config read failure
14:40	Root cause identified: missing DATABASE_URL env var
14:45	Rollback initiated
15:10	Error rate returns to baseline

Root cause: New environment variable added to code but not to docker-compose.yml.

Detection gap: 6-minute lag between incident start and alert firing. Action item: reduce fast-burn for: clause from 2m to 1m.

Action items:

Action	Owner	Due
Add post-deploy smoke test	DevOps	3 days
Add env var validation to entrypoint	App dev	5 days
Reduce fast-burn for: clause to 1m	DevOps	1 day

This review is blameless — we focus on systems and processes, not individuals.

Part 8: Game Day Results

Scenario 1 — Deployment Failure

📸 [Screenshot: GitHub Actions showing red failed run]

📸 [Screenshot: CICDDeploymentFailed in Slack]

Scenario 2 — Latency Injection

Injected 600ms network latency:

sudo tc qdisc add dev ens5 root netem delay 600ms

HighLatencyWarning fired confirming the alerting pipeline for latency SLO breaches works end-to-end.

# Remove latency
sudo tc qdisc del dev ens5 root

RESOLVED message confirmed recovery detection works.

📸 [Screenshot: Unified dashboard showing latency spike]

📸 [Screenshot: HighLatencyWarning in Slack]

📸 [Screenshot: RESOLVED in Slack after tc removed]

Scenario 3 — Resource Pressure

Used stress-ng to drive CPU above 90%:

stress-ng --cpu 0 --cpu-method matrixprod --timeout 600s &

What we observed:

HighCPUWarning entered pending state after CPU sustained above 80%
After 5 minutes → HighCPUWarning turned firing in Prometheus
Alert arrived in Slack with full structured payload
HighCPUCritical entered pending (needs 10min sustained above 90%)
After killing stress: both alerts RESOLVED in Slack

This confirmed the full warning → critical → recovery sequence and proved inhibition rules work — critical suppressed the warning notification.

pkill stress-ng

📸 [Screenshot: Prometheus alerts page showing Warning firing]

📸 [Screenshot: Prometheus alerts page showing Critical pending]

📸 [Screenshot: Node Exporter dashboard with CPU spike at 92%]

📸 [Screenshot: HighCPUWarning in Slack]

📸 [Screenshot: RESOLVED in Slack]

Key Learnings

1. Observability is not monitoring.
Monitoring tells you something is wrong. Observability tells you why, where, and when — without needing to SSH into a server.

2. SLOs make reliability decisions objective.
"Is this deployment safe?" is subjective. "Do we have 100 minutes of error budget remaining?" is objective. SLOs turn reliability from a conversation into a measurement.

3. Burn rate alerting eliminates alert fatigue.
Two burn rate alerts replaced what would have been dozens of threshold alerts during our Game Day scenarios. Engineers respond to meaningful signals, not noise.

4. DORA metrics connect engineering to business.
High MTTR isn't just a technical problem — it's lost revenue per minute. Low deployment frequency isn't just slow — it's delayed value delivery. DORA makes this explicit.

5. Everything as code is non-negotiable.
Every dashboard, alert rule, and config that lives only in a UI is technical debt. When the server dies, you want to run docker compose up -d and have everything back — not spend three hours recreating dashboards from memory.

Conclusion

The MeetMind Observability Platform demonstrates that production-grade observability is achievable without managed services. The LGTM stack provides the full observability triad — metrics, logs, and traces — with correlation between all three. SLOs convert vague reliability goals into measurable targets. DORA metrics connect daily engineering decisions to business outcomes. Burn rate alerting replaces alert storms with two meaningful signals.

The entire platform deploys with one command. Every component is version-controlled. Every alert links to a runbook. Every metric spike links to correlated logs and traces.

GitHub Repository: https://github.com/AirFluke/meetmind-observability

Built by Team MeetMind for HNG DevOps Track Stage 6

SwiftDeploy: Building a Self-Writing Infrastructure Manager with Policy Enforcement — A Complete Technical Walkthrough

Abraham Acha — Wed, 06 May 2026 04:01:09 +0000

How I built a CLI tool that generates its own infrastructure configs, manages a full containerised stack, enforces deployment policies through OPA, exposes Prometheus metrics, and produces a live audit trail — all from a single YAML file.

The Problem We're Solving
Architecture Overview
Stage 4A — The Engine
- The Manifest
- The Python HTTP Service
- The Dockerfile
- The Templates
- The CLI
- The Two Deployment Modes
Stage 4B — The Eyes and Brain
- Prometheus Metrics
- OPA Policy Engine
- Gated Lifecycle
- The Status Dashboard
- The Audit Trail
The Debugging Sagas
Full Deployment Walkthrough
Key Lessons Learned

The Problem We're Solving {#the-problem}

Every time you spin up a new service in a real DevOps environment, you repeat the same manual work:

Write an Nginx config
Write a Docker Compose file
Run Docker commands
Check if things are healthy
Hope nobody deploys when the disk is full
Hope nobody promotes a canary that's throwing 60% errors

SwiftDeploy solves all of this. One YAML manifest describes your entire deployment. A CLI tool generates every config file from it, manages the container lifecycle, enforces safety policies before allowing deployments, exposes real-time metrics, and produces a full audit trail.

The golden rule: the manifest is the single source of truth. Everything else is generated.

Architecture Overview {#architecture}

┌─────────────────────────────────────────────────────────────────────────────┐
│                    SwiftDeploy — Full System Architecture                   │
├──────────────────┬──────────────────────────────────────┬───────────────────┤
│   ZONE 1         │          ZONE 2                      │   ZONE 3          │
│   Operator       │     Host Machine / Docker Engine     │   Generated Files │
│                  │                                      │                   │
│  [Operator]      │  ┌─── swiftdeploy-net (bridge) ───┐  │  nginx.conf       │
│      │           │  │                                │  │  docker-compose   │
│      ▼           │  │  [nginx:8080]──────►[app:3000] │  │  history.jsonl    │
│  manifest.yaml   │  │   PUBLIC           INTERNAL    │  │  audit_report.md  │
│  (source of      │  │      │                 │       │  │                   │
│   truth)         │  │      └──[logs vol]─────┘       │  │                   │
│      │           │  │                                │  │                   │
│      ▼           │  │  [opa:8181]                    │  │                   │
│  swiftdeploy     │  │  localhost only                │  │                   │
│  CLI             │  │  NOT via nginx ✗               │  │                   │
│  ├─ init         │  └────────────────────────────────┘  │                   │
│  ├─ validate     │                                      │                   │
│  ├─ deploy ──────┼──► pre-deploy: OPA infra check       │                   │
│  ├─ promote ─────┼──► pre-promote: OPA canary check     │                   │
│  ├─ status ──────┼──► scrapes /metrics every 5s ───────►│ history.jsonl     │
│  ├─ audit ───────┼────────────────────────────────────►│ audit_report.md   │
│  └─ teardown     │                                      │                   │
└──────────────────┴──────────────────────────────────────┴───────────────────┘

Stage 4A — The Engine {#stage-4a}

Stage 4A is the foundation. It answers one question: how do you build a tool that writes its own infrastructure configs?

The Project Structure

swiftdeploy/
├── manifest.yaml                    ← the ONLY file you edit
├── swiftdeploy                      ← CLI executable
├── Dockerfile                       ← app image definition
├── app/
│   └── main.py                      ← Python HTTP service
├── templates/
│   ├── nginx.conf.tmpl              ← nginx template
│   └── docker-compose.yml.tmpl     ← compose template
├── policies/                        ← Stage 4B addition
│   ├── infrastructure.rego
│   ├── canary.rego
│   └── data.json
├── nginx.conf                       ← generated (gitignored)
└── docker-compose.yml               ← generated (gitignored)

The Manifest {#the-manifest}

manifest.yaml is the brain of the entire system. Every component reads from it directly or via generated files.

services:
  image: swift-deploy-1-node:latest
  port: 3000
  mode: stable                    # stable or canary
  version: "1.0.0"
  restart_policy: unless-stopped
  log_volume: swiftdeploy-logs

nginx:
  image: nginx:latest
  port: 8080
  proxy_timeout: 30

opa:
  image: openpolicyagent/opa:latest-static
  port: 8181

network:
  name: swiftdeploy-net
  driver_type: bridge

contact: "ops@swiftdeploy.local"

Every field propagates through the system. Change nginx.proxy_timeout here and it updates in nginx.conf on the next init. Change services.mode here and the entire deployment mode switches on the next promote.

The Python HTTP Service {#the-python-service}

The app is a from-scratch HTTP server using only Python's stdlib — no Flask, no FastAPI. Three endpoints in Stage 4A, four in Stage 4B.

Configuration from environment

MODE        = os.environ.get("MODE", "stable")
APP_VERSION = os.environ.get("APP_VERSION", "1.0.0")
APP_PORT    = int(os.environ.get("APP_PORT", "3000"))
START_TIME  = time.time()

Configuration comes entirely from environment variables injected by Docker Compose at runtime. START_TIME is captured at module load — this is how /healthz calculates uptime without a database.

Thread-safe chaos state

chaos_lock = threading.Lock()
chaos_state = {"mode": None, "duration": None, "rate": None}

def get_chaos():
    with chaos_lock:
        return dict(chaos_state)  # returns a copy — callers can't mutate internal state

def set_chaos(state):
    with chaos_lock:
        chaos_state.update(state)

The Lock prevents race conditions when multiple requests read and write chaos state simultaneously. dict(chaos_state) returns a copy so the caller never holds a reference to the mutable internal dict.

The three Stage 4A endpoints

GET / — welcome

self.send_json(200, {
    "message": "Welcome to SwiftDeploy API",
    "mode": MODE,
    "version": APP_VERSION,
    "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
})

GET /healthz — liveness check

uptime = round(time.time() - START_TIME, 2)
self.send_json(200, {
    "status": "ok",
    "mode": MODE,
    "version": APP_VERSION,
    "uptime_seconds": uptime,
})

The /healthz endpoint does three jobs: proves the server is alive (Docker healthcheck), reports current mode (so promote can confirm the switch happened), and reports uptime (useful for debugging restart loops).

POST /chaos — chaos injection (canary only)

if MODE != "canary":
    self.send_json(403, {"error": "chaos endpoint only available in canary mode"})
    return

length = int(self.headers.get("Content-Length", 0))
data   = json.loads(self.rfile.read(length))
mode   = data.get("mode")

if mode == "slow":
    set_chaos({"mode": "slow", "duration": data.get("duration", 2), "rate": None})
elif mode == "error":
    set_chaos({"mode": "error", "duration": None, "rate": data.get("rate", 0.5)})
elif mode == "recover":
    set_chaos({"mode": None, "duration": None, "rate": None})

Reading Content-Length before rfile.read() is mandatory HTTP protocol — otherwise read() blocks forever waiting for data that never arrives.

The chaos modes:

slow — injects time.sleep(N) before responding, simulating a slow upstream
error — uses random.random() < rate to return 500 on a configurable percentage of requests
recover — clears all chaos state, returning to normal behaviour

The Dockerfile {#the-dockerfile}

FROM python:3.12-alpine

RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app
COPY app/main.py .
RUN chown -R appuser:appgroup /app

USER appuser

ENV MODE=stable
ENV APP_VERSION=1.0.0
ENV APP_PORT=3000

EXPOSE 3000

HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=5 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:3000/healthz', timeout=4)" || exit 1

CMD ["python", "main.py"]

Why Alpine? python:3.12-alpine is ~60MB. python:3.12 (Debian) is ~1GB. We need under 300MB.

Why non-root? If someone exploits the app, they get a powerless user, not root access to the server.

Why Python urllib for the healthcheck? This was a hard-won lesson. wget couldn't resolve localhost inside Alpine on WSL2 + Docker Desktop — a known network namespace quirk. Python's urllib bypasses the system resolver entirely and connects directly via socket. More reliable, no external tool needed.

The Templates {#the-templates}

Templates are blueprints with {{ placeholder }} values that the CLI replaces with real values from the manifest.

nginx.conf.tmpl key sections:

upstream app_backend {
    server app:{{ service_port }};
    keepalive 32;
}

log_format swiftdeploy '$time_iso8601 | $status | ${request_time}s | $upstream_addr | $request';

server {
    listen {{ nginx_port }};

    proxy_connect_timeout {{ proxy_timeout }}s;
    proxy_send_timeout    {{ proxy_timeout }}s;
    proxy_read_timeout    {{ proxy_timeout }}s;

    add_header X-Deployed-By swiftdeploy always;
    proxy_pass_header X-Mode;

    location @error502 {
        default_type application/json;
        return 502 '{"error":"Bad Gateway","code":502,"service":"app","contact":"{{ contact }}"}';
    }
}

Design decisions:

Custom log format — ISO timestamp, status code, response time, upstream IP, full request on one line
JSON error bodies — APIs need machine-readable errors, not nginx HTML pages
proxy_pass_header X-Mode — nginx strips custom headers by default; this forwards the canary header through to clients
keepalive 32 — maintains 32 persistent connections to the upstream, reducing connection overhead

docker-compose.yml.tmpl key sections:

app:
  expose:
    - "{{ service_port }}"    # container-to-container only, NEVER published to host

nginx:
  ports:
    - "{{ nginx_port }}:{{ nginx_port }}"    # only nginx faces the world

  depends_on:
    app:
      condition: service_healthy    # nginx waits for app healthcheck to pass

The expose vs ports distinction is a security boundary. expose = container-to-container only. ports = host-facing. The app is never reachable from outside Docker.

The CLI — Five Stage 4A Subcommands {#the-cli}

The template engine

def render_template(tmpl_path, context):
    with open(tmpl_path) as f:
        content = f.read()
    for key, val in context.items():
        content = content.replace("{{ " + key + " }}", str(val))
    return content

Five lines. No Jinja2. Simple string replacement. The templates are straightforward enough that a minimal custom engine is cleaner than pulling in a dependency.

`init` — generates everything from the manifest

def cmd_init():
    manifest = load_manifest()
    ctx      = build_context(manifest)

    nginx_conf   = render_template(NGINX_TMPL, ctx)
    compose_conf = render_template(COMPOSE_TMPL, ctx)

    with open(NGINX_OUT, "w") as f:   f.write(nginx_conf)
    with open(COMPOSE_OUT, "w") as f: f.write(compose_conf)

The grader deletes generated files and re-runs init to verify regeneration. Because we built it correctly, this is a non-issue.

`validate` — five pre-flight checks

# Check 1: manifest.yaml exists and is valid YAML
# Check 2: all required fields present and non-empty
# Check 3: docker image inspect — exits 0 if exists
# Check 4: ss -tlnp | grep :8080 — non-empty means port in use
# Check 5: nginx -t via isolated Docker container

Check 5 is the most interesting. We run nginx -t in a temporary container — validating syntax without needing nginx installed on the host. But we hit a complication: app:3000 (the upstream hostname) can't be resolved in an isolated container. Fix: swap server app: with server 127.0.0.1: in a temp copy before testing. Actual nginx.conf on disk is untouched.

test_content = data.replace("server app:", "server 127.0.0.1:")

`deploy` — brings up the stack and blocks until healthy

deadline = time.time() + 60
while time.time() < deadline:
    try:
        with urllib.request.urlopen(f"http://localhost:{nginx_port}/healthz", timeout=3) as resp:
            if json.loads(resp.read()).get("status") == "ok":
                healthy = True
                break
    except Exception:
        pass    # container still starting — connection refused is normal
    time.sleep(2)

docker compose up -d returns as soon as containers are created, not when they're healthy. The polling loop tries /healthz every 2 seconds for 60 seconds. Only {"status": "ok"} breaks the loop.

`promote` — rolling restart with zero nginx downtime

# 1. Update manifest in-place
content = re.sub(r"(mode:\s*)(\S+)", f"\\g<1>{target_mode}", content, count=1)

# 2. Regenerate docker-compose.yml with new MODE env var

# 3. Restart ONLY the app container — nginx stays up
run(compose_cmd("up -d --no-deps app"))

# 4. Confirm mode via /healthz

--no-deps is the key — it tells Compose to restart only the app service without touching nginx. Zero proxy downtime during the switch.

The Two Deployment Modes {#deployment-modes}

Stable mode — normal production behaviour. Clean responses. No special headers. /chaos returns 403.

Canary mode — test mode before full rollout:

if MODE == "canary":
    self.send_header("X-Mode", "canary")    # on EVERY response

Canary mode:

Adds X-Mode: canary to every response — callers can identify which mode they're hitting
Unlocks /chaos — lets you simulate slow responses, random errors, then recover
You promote with ./swiftdeploy promote canary, stress test, then ./swiftdeploy promote stable to roll back

Stage 4B — The Eyes and The Brain {#stage-4b}

Stage 4A built the engine. Stage 4B adds observability and policy enforcement. The stack now has eyes (metrics), a brain (OPA policies), and memory (audit trail).

Prometheus Metrics — The Eyes {#prometheus-metrics}

The app gains a /metrics endpoint exposing five metric types in Prometheus text format.

Tracking infrastructure

# Counter: {(method, path, status_code): count}
request_counts = {}

# Histogram state per path
HISTOGRAM_BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
request_durations = {}

def record_request(method, path, status_code, duration_seconds):
    with metrics_lock:
        key = (method, path, str(status_code))
        request_counts[key] = request_counts.get(key, 0) + 1

        if path not in request_durations:
            request_durations[path] = {
                "buckets": {str(le): 0 for le in HISTOGRAM_BUCKETS},
                "sum": 0.0,
                "count": 0,
            }
        hist = request_durations[path]
        hist["sum"] += duration_seconds
        hist["count"] += 1
        for le in HISTOGRAM_BUCKETS:
            if duration_seconds <= le:
                hist["buckets"][str(le)] += 1

record_request() is called after every request regardless of path — timing wraps the entire handler:

def do_GET(self):
    start  = time.time()
    path   = self.path.split("?")[0]
    status = self._handle_get()
    record_request("GET", path, status, time.time() - start)

The five metrics

# 1. http_requests_total — counter, labels: method, path, status_code
# 2. http_request_duration_seconds — histogram with standard buckets
# 3. app_uptime_seconds — gauge
# 4. app_mode — gauge: 0=stable, 1=canary
# 5. chaos_active — gauge: 0=none, 1=slow, 2=error

The /metrics output

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/",status_code="200"} 42
http_requests_total{method="GET",path="/healthz",status_code="200"} 60
http_requests_total{method="GET",path="/",status_code="500"} 38

# HELP http_request_duration_seconds HTTP request latency in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{path="/",le="0.005"} 40
http_request_duration_seconds_bucket{path="/",le="+Inf"} 80
http_request_duration_seconds_sum{path="/"} 0.042381
http_request_duration_seconds_count{path="/"} 80

# HELP app_mode Current deployment mode (0=stable, 1=canary)
# TYPE app_mode gauge
app_mode 1

# HELP chaos_active Current chaos state (0=none, 1=slow, 2=error)
# TYPE chaos_active gauge
chaos_active 2

OPA Policy Engine — The Brain {#opa-policy-engine}

Open Policy Agent is a dedicated container whose only job is making allow/deny decisions based on rules written in Rego. The core principle: the CLI never makes allow/deny decisions itself. All decision logic lives exclusively in OPA.

Why OPA instead of if/else in the CLI?

If the policy logic lives in the CLI, changing a threshold means editing Python code, rebuilding, redeploying. With OPA, you edit data.json, restart the OPA container, and the new threshold is live. Policy as code, not policy as application logic.

data.json — thresholds live here, never hardcoded in Rego

{
  "thresholds": {
    "min_disk_free_gb": 10.0,
    "max_cpu_load": 2.0,
    "min_mem_free_percent": 10.0,
    "max_error_rate_percent": 1.0,
    "max_p99_latency_ms": 500.0
  }
}

infrastructure.rego — pre-deploy policy

package swiftdeploy.infrastructure

import future.keywords.if
import future.keywords.contains

default allow := false

allow if {
    disk_ok
    cpu_ok
    mem_ok
}

disk_ok if { input.disk_free_gb >= data.thresholds.min_disk_free_gb }
cpu_ok  if { input.cpu_load_1m  <= data.thresholds.max_cpu_load }
mem_ok  if { input.mem_free_percent >= data.thresholds.min_mem_free_percent }

reasons contains msg if {
    not disk_ok
    msg := sprintf(
        "disk_free_gb is %.1f, minimum required is %.1f",
        [input.disk_free_gb, data.thresholds.min_disk_free_gb]
    )
}

# decision is what the CLI reads — never a bare boolean
decision := {
    "allow":   allow,
    "reasons": reasons,
    "domain":  "infrastructure",
    "checked": {
        "disk_free_gb":     input.disk_free_gb,
        "cpu_load_1m":      input.cpu_load_1m,
        "mem_free_percent": input.mem_free_percent,
    },
}

Why import future.keywords? The openpolicyagent/opa:latest-static image uses Rego v1 which requires explicit if and contains keywords. Without these imports, OPA crashes on startup. This was discovered the hard way during testing.

canary.rego — pre-promote policy

package swiftdeploy.canary

import future.keywords.if
import future.keywords.contains

default allow := false

allow if {
    error_rate_ok
    latency_ok
}

error_rate_ok if { input.error_rate_percent <= data.thresholds.max_error_rate_percent }
latency_ok    if { input.p99_latency_ms     <= data.thresholds.max_p99_latency_ms }

reasons contains msg if {
    not error_rate_ok
    msg := sprintf(
        "error_rate is %.2f%%, maximum allowed is %.2f%%",
        [input.error_rate_percent, data.thresholds.max_error_rate_percent]
    )
}

decision := {
    "allow":   allow,
    "reasons": reasons,
    "domain":  "canary",
    "checked": {
        "error_rate_percent": input.error_rate_percent,
        "p99_latency_ms":     input.p99_latency_ms,
        "window_seconds":     input.window_seconds,
    },
}

OPA isolation — no leakage via nginx

In docker-compose.yml.tmpl:

opa:
  ports:
    - "127.0.0.1:{{ opa_port }}:8181"    # localhost only — never 0.0.0.0

127.0.0.1:8181 means only the host machine can reach OPA. The nginx container on port 8080 has no route to OPA. This is enforced at the Docker network binding level, not just convention.

The policy query function

def query_opa(manifest, package, input_data):
    url     = f"{opa_url(manifest)}/v1/data/{package.replace('.', '/')}/decision"
    payload = json.dumps({"input": input_data}).encode()
    try:
        req = urllib.request.Request(url, data=payload,
              headers={"Content-Type": "application/json"}, method="POST")
        with urllib.request.urlopen(req, timeout=5) as resp:
            body   = json.loads(resp.read())
            result = body.get("result")
            if result is None:
                return None, "OPA returned empty result — check policy package name"
            return result, None
    except urllib.error.URLError as e:
        return None, f"OPA unreachable: {e.reason}"
    except Exception as e:
        return None, f"OPA query failed: {e}"

Every distinct failure mode returns a different error string. The CLI never crashes or hangs when OPA is unavailable — it warns and fails open. This is intentional: you don't want OPA unavailability to block emergency deployments.

Gated Lifecycle — The CLI Brain {#gated-lifecycle}

Pre-deploy check

def cmd_deploy():
    manifest   = load_manifest()
    host_stats = get_host_stats()

    # Collect host stats
    disk_free_gb     = shutil.disk_usage("/").free / (1024 ** 3)
    cpu_load_1m      = float(open("/proc/loadavg").read().split()[0])
    mem_free_percent = (meminfo["MemAvailable"] / meminfo["MemTotal"]) * 100

    # Send to OPA
    allowed = enforce_policy(manifest, "swiftdeploy.infrastructure", host_stats, "infrastructure")

    if not allowed:
        append_history({"event": "deploy_blocked", "reason": "infrastructure_policy"})
        sys.exit(1)

    # Only reach here if OPA allows
    run(compose_cmd("up -d --build"))

If the disk is full, OPA returns:

  ✘ Policy [infrastructure] DENIED
    ! disk_free_gb is 8.2, minimum required is 10.0

  Deployment blocked by policy: infrastructure

Pre-promote check

def cmd_promote(target_mode):
    if target_mode == "canary":
        raw        = scrape_metrics(nginx_port)
        metrics    = parse_prometheus(raw)
        error_rate = calculate_error_rate(metrics)
        p99_ms     = calculate_p99_latency_ms(metrics)

        allowed = enforce_policy(manifest, "swiftdeploy.canary",
            {"error_rate_percent": error_rate, "p99_latency_ms": p99_ms, "window_seconds": 30},
            "canary safety"
        )
        if not allowed:
            sys.exit(1)

P99 latency calculation from histogram

def calculate_p99_latency_ms(metrics, path_filter=None):
    buckets     = {}
    total_count = 0

    for entry in metrics.get("http_request_duration_seconds_bucket", []):
        le = entry["labels"].get("le", "")
        if le == "+Inf":
            total_count = max(total_count, entry["value"])
            continue
        buckets[float(le)] = buckets.get(float(le), 0) + entry["value"]

    if total_count == 0:
        return 0.0

    p99_threshold = total_count * 0.99
    for le in sorted(buckets.keys()):
        if buckets[le] >= p99_threshold:
            return round(le * 1000, 2)    # seconds → milliseconds
    return 10000.0

P99 means: the smallest histogram bucket where 99% of requests have completed. If 99 out of 100 requests finished within 250ms, P99 = 250ms.

The Status Dashboard — The Eyes {#status-dashboard}

def cmd_status():
    while True:
        raw        = scrape_metrics(nginx_port)
        metrics    = parse_prometheus(raw)
        error_rate = calculate_error_rate(metrics)
        p99_ms     = calculate_p99_latency_ms(metrics)

        # Query OPA for live compliance
        infra_dec,  _ = query_opa(manifest, "swiftdeploy.infrastructure", get_host_stats())
        canary_dec, _ = query_opa(manifest, "swiftdeploy.canary",
                            {"error_rate_percent": error_rate, "p99_latency_ms": p99_ms, "window_seconds": 30})

        os.system("clear")
        # ... render dashboard ...

        append_history({
            "event":              "status_scrape",
            "error_rate_percent": error_rate,
            "p99_latency_ms":     p99_ms,
            "mode":               mode_str,
            "chaos":              chaos_str,
            "policy_infra_pass":  infra_dec.get("allow") if infra_dec else None,
            "policy_canary_pass": canary_dec.get("allow") if canary_dec else None,
        })

        time.sleep(5)

What the dashboard looks like with chaos active:

  SwiftDeploy Status Dashboard  2026-05-05T21:00:37Z
  ────────────────────────────────────────────────

  ── Throughput ──────────────────────────────────
  req/s       : 2.4
  error rate  : 56.45%      ← red
  P99 latency : 5.0ms

  ── App State ───────────────────────────────────
  mode        : canary
  chaos       : error       ← red
  uptime      : 316s

  ── Policy Compliance ───────────────────────────
  ✔  infrastructure       PASS
  ✘  canary               FAIL
       ! error_rate is 56.45%, maximum allowed is 1.00%

  Refreshing every 5s — Ctrl+C to exit

This is exactly what real SRE dashboards do — they show you the current state AND whether it violates policy in real time.

The Audit Trail — The Memory {#audit-trail}

Every event appends a JSON line to history.jsonl:

{"timestamp":"2026-05-05T20:34:51Z","event":"deploy","mode":"stable"}
{"timestamp":"2026-05-05T20:55:22Z","event":"promote","target_mode":"canary"}
{"timestamp":"2026-05-05T20:55:23Z","event":"status_scrape","error_rate_percent":62.5,"chaos":"error","policy_canary_pass":false}
{"timestamp":"2026-05-05T21:01:01Z","event":"promote","target_mode":"stable"}

swiftdeploy audit reads this file and generates audit_report.md:

## Mode Changes

| Timestamp | From | To |
|-----------|------|----|
| 2026-05-05T20:34:51Z | unknown | stable |
| 2026-05-05T20:55:22Z | stable  | canary |
| 2026-05-05T21:01:01Z | canary  | stable |

## Policy Violations

| Timestamp | Infrastructure | Canary | Error Rate | P99 |
|-----------|---------------|--------|------------|-----|
| 2026-05-05T20:55:23Z | ✔ PASS | ✘ FAIL | 62.5% | 5.0ms |
| 2026-05-05T20:55:28Z | ✔ PASS | ✘ FAIL | 63.6% | 5.0ms |

This report renders perfectly as GitHub Flavored Markdown — every table, every checkmark, every timestamp.

The Debugging Sagas {#debugging}

No real DevOps project ships without war stories. Here are the ones that taught the most.

Saga 1 — Six layers of healthcheck failure

The app container kept showing unhealthy despite the server running fine. The debugging sequence:

Failure 1: ${APP_PORT} doesn't expand in Dockerfile HEALTHCHECK CMD. Env vars evaluate at build time, not runtime. Fixed by hardcoding 3000.

Failure 2: localhost doesn't resolve inside Alpine's healthcheck context on WSL2 + Docker Desktop. Fixed by using 127.0.0.1.

Failure 3: wget with 127.0.0.1 still failed. Confirmed the server WAS listening:

docker exec swiftdeploy-app ss -tlnp
# tcp LISTEN 0.0.0.0:3000
docker exec swiftdeploy-app python -c "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:3000/healthz').read())"
# b'{"status": "ok", ...}'   ← works via exec, not via healthcheck

This is a known WSL2 + Docker Desktop network namespace issue. Fixed by using Python's urllib instead of wget.

Failure 4: Docker cache was serving the old image despite the Dockerfile fix. Fixed with --no-cache.

Failure 5: The docker-compose.yml template had its own healthcheck block overriding the Dockerfile. Docker Compose healthcheck always wins. Fixed the template too.

Failure 6: The healthcheck YAML block had 3 spaces indent instead of 4. A single space difference caused a YAML parse error. Fixed by carefully rewriting the block.

Saga 2 — OPA Rego v1 syntax

The openpolicyagent/opa:latest-static image enforces strict Rego v1 syntax. Our policies used the older syntax:

# OLD — crashes on latest OPA
allow {
    disk_ok
}
reasons[msg] {
    not disk_ok
    msg := "..."
}

# NEW — Rego v1 required syntax
allow if {
    disk_ok
}
reasons contains msg if {
    not disk_ok
    msg := "..."
}

Without import future.keywords.if and import future.keywords.contains at the top of each file, OPA refuses to start.

Saga 3 — WSL2 path spaces breaking docker run

The project lived at /mnt/c/Users/RAZER BLADE/Desktop/HNG/hng-swiftdeploy. The space in RAZER BLADE caused docker run -v {path}:... to split the path at the space, making Docker interpret the second half as an image name.

docker: invalid reference format: repository name
(Desktop/HNG/hng-swiftdeploy/nginx.conf) must be lowercase

Fixed by quoting all paths containing the project directory and using subprocess.run with a list instead of shell=True to avoid shell word-splitting entirely.

Full Deployment Walkthrough {#deployment}

# 1. Build the image
docker build -t swift-deploy-1-node:latest .

# 2. Validate pre-flight checks
./swiftdeploy validate

# 3. Deploy (OPA policy check runs first)
./swiftdeploy deploy

# 4. Verify metrics
curl http://localhost:8080/metrics

# 5. Verify OPA isolation
curl http://127.0.0.1:8181/health    # works — internal
curl http://localhost:8080/v1/data   # 404 — nginx blocks it

# 6. Launch status dashboard
./swiftdeploy status

# 7. Promote to canary (OPA canary policy check runs first)
./swiftdeploy promote canary

# 8. Inject chaos
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "error", "rate": 0.5}'

# 9. Watch status dashboard catch it — canary policy FAIL visible in real time

# 10. Recover
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode": "recover"}'

# 11. Promote back to stable
./swiftdeploy promote stable

# 12. Generate audit report
./swiftdeploy audit
cat audit_report.md

# 13. Teardown
./swiftdeploy teardown --clean

Key Lessons Learned {#lessons}

1. Docker Compose healthcheck overrides Dockerfile HEALTHCHECK. Always check both places when healthchecks misbehave. Compose wins every time.

2. WSL2 has a different network namespace for healthchecks than for docker exec. If something works via exec but not via healthcheck, it's almost certainly a tool or network namespace issue. Python's stdlib is more portable than wget in this environment.

3. OPA Rego v1 requires explicit keywords. latest-static means the latest OPA — which enforces Rego v1 syntax. Always import future.keywords.if and import future.keywords.contains.

4. expose vs ports is a security boundary, not documentation. expose = container-to-container only. ports = host-facing. Binding OPA to 127.0.0.1 enforces isolation at the network level.

5. The CLI should never make policy decisions. Every time you add an if/else for a deployment condition in the CLI, you're doing OPA's job badly. Push all allow/deny logic into Rego. The CLI's job is to collect data and surface decisions.

6. P99 latency is more useful than average. An average latency of 10ms can hide the fact that 1 in 100 requests takes 5 seconds. P99 exposes that tail. Always instrument histograms, not just averages.

7. Declarative infrastructure pays off immediately. The grader deletes generated files and re-runs init. Because the manifest is always there and regeneration is instantaneous, this is a non-issue. Manual configs would have been a problem.

8. An audit trail is not optional. history.jsonl made it trivial to answer "when did chaos start?", "which policy was failing?", "how long was the canary running before we promoted?" These questions matter in production incidents.

Conclusion

SwiftDeploy started as a task requirement and became a complete mental model for how modern deployment tooling works. Every major concept is here:

Declarative infrastructure — describe what you want, generate everything else
Immutable configs — generated files are outputs, never inputs
Policy as code — OPA enforces safety standards that can't be bypassed
Observability — Prometheus metrics feed the dashboard and the policy engine
Audit trail — every event recorded, every violation surfaced

The combination of Stage 4A and 4B forms a complete deployment lifecycle: generate → validate → deploy (gated) → promote (gated) → observe → audit → tear down.

The full source code is available at: https://github.com/AirFluke/hng-swiftdeploy

Tags: #devops #docker #nginx #python #opa #prometheus #infrastructure #hng

How I Built a Real-Time DDoS Detection Engine from Scratch (No Fail2Ban)

Abraham Acha — Sun, 26 Apr 2026 15:31:10 +0000

Assuming my boss walked in and said "we're getting hit with suspicious traffic and I need you to build something that detects and responds to it automatically," I had two choices: reach for Fail2Ban like everyone else, or build something I actually understood from the ground up.

I chose to build it from scratch. This post explains exactly how I did it — in plain English, with real code, and without assuming you've ever touched security tooling before.

By the end of this post you will understand:

What the system does and why it matters
How a sliding window works (and why it's not just "count requests per minute")
How the system learns what normal traffic looks like
How it decides something is an attack
How it uses iptables to cut off an attacker at the firewall level

Let's go.

What This Project Does and Why It Matters

We run a cloud storage platform built on Nextcloud. It's public-facing, which means anyone on the internet can send requests to it — including bots, scanners, and attackers.

Two types of attacks concern us most:

A single aggressive IP — one machine sending hundreds of requests per second, trying to brute-force logins, scrape data, or simply overwhelm the server. This is a classic DDoS from a single source.

A global traffic spike — thousands of different IPs each sending a small number of requests. No single IP looks suspicious, but the total volume is crushing the server. This is a distributed DDoS.

The tool I built watches every HTTP request in real time, learns what "normal" traffic looks like, and fires an automatic response the moment something deviates — whether it's a single aggressive IP or a global spike.

For a single aggressive IP: it adds an iptables firewall rule to drop every packet from that IP, sends a Slack alert, and automatically releases the ban on a backoff schedule (10 minutes, then 30, then 2 hours, then permanent for repeat offenders).

For a global spike: it sends a Slack alert (no single IP to block).

The whole thing runs as a daemon — a continuously running background process — alongside the Nextcloud stack in Docker. It never sleeps, never needs a cron job, and never relies on hardcoded thresholds.

Here's the architecture at a glance:

Internet → Nginx (reverse proxy) → Nextcloud
                ↓ writes JSON logs
         [HNG-nginx-logs volume]
                ↓ reads logs
         Python Daemon
           ├── monitor.py    (tail the log file)
           ├── baseline.py   (learn normal traffic)
           ├── detector.py   (spot anomalies)
           ├── blocker.py    (iptables + audit log)
           ├── unbanner.py   (auto-release bans)
           ├── notifier.py   (Slack alerts)
           └── dashboard.py  (live web UI)

Everything is wired together by main.py and configured through a single config.yaml file — no hardcoded numbers anywhere in the code.

Part 1: Reading the Traffic — The Log Monitor

Before we can detect anything, we need to see the traffic. Nginx is our reverse proxy — every HTTP request passes through it. We configure Nginx to write each request as a JSON object to a log file.

Here's the Nginx log format we defined:

log_format hng_json escape=json
    '{'
        '"source_ip":"$remote_addr",'
        '"timestamp":"$time_iso8601",'
        '"method":"$request_method",'
        '"path":"$uri",'
        '"status":$status,'
        '"response_size":$body_bytes_sent,'
        '"request_time":$request_time,'
        '"http_user_agent":"$http_user_agent"'
    '}';

access_log /var/log/nginx/hng-access.log hng_json;

Every request now produces a line like this in the log file:

{
  "source_ip": "1.2.3.4",
  "timestamp": "2026-04-26T14:08:21+00:00",
  "method": "GET",
  "path": "/index.php/login",
  "status": 401,
  "response_size": 1842,
  "request_time": 0.043,
  "http_user_agent": "Mozilla/5.0..."
}

Now our Python daemon needs to read these lines as they appear — in real time, without missing any, and without crashing if the log file gets rotated. This is called "tailing" a file, like tail -f in Linux.

Here's how monitor.py does it:

import os
import time
import json
import queue

def parse_line(raw: str) -> dict | None:
    """Parse one JSON log line. Returns None if malformed."""
    raw = raw.strip()
    if not raw:
        return None
    try:
        entry = json.loads(raw)
    except json.JSONDecodeError:
        return None  # drop malformed lines silently

    # Make sure all required fields are present
    required = {"source_ip", "timestamp", "method", "path", "status", "response_size"}
    if required - entry.keys():
        return None

    # Coerce types and add a wall-clock timestamp for window math
    entry["status"] = int(entry["status"])
    entry["response_size"] = int(entry["response_size"])
    entry["_parsed_at"] = time.time()
    return entry


class LogMonitor:
    def run(self):
        fh = open("/var/log/nginx/hng-access.log", "r")
        fh.seek(0, 2)  # jump to END of file — don't replay old traffic

        current_inode = os.stat(fh.name).st_ino

        while True:
            line = fh.readline()
            if line:
                entry = parse_line(line)
                if entry:
                    self.out_queue.put(entry)  # send downstream
            else:
                # Check if the file was rotated (inode changed)
                new_inode = os.stat(fh.name).st_ino
                if new_inode != current_inode:
                    fh.close()
                    fh = open(fh.name, "r")
                    current_inode = new_inode
                else:
                    time.sleep(0.1)  # no new data, yield CPU briefly

Three things worth understanding here:

fh.seek(0, 2) — When we start the daemon, the log file already has entries from before we launched. We don't want to process all of those as if they're live traffic. Seeking to position 2 (the end) means we only see new lines written after the daemon starts.

Inode check — Log files get rotated (renamed and replaced with a fresh empty file) periodically. If we don't detect this, we'd keep reading the old file and miss all new traffic. The inode (a unique file identifier in Linux) changes when the file is replaced, so we check it on every "no new data" cycle.

The queue — We put parsed entries into a queue.Queue. This decouples reading from processing. The monitor thread just reads and queues; the main thread drains the queue and feeds entries to the baseline and detector. If the detector is temporarily slow, entries buffer in the queue rather than being dropped.

Part 2: How the Sliding Window Works

This is the core data structure of the entire detection system. Get this right and everything else falls into place.

The wrong way: per-minute counters

The naive approach is to count requests per minute:

# WRONG — this is not a sliding window
requests_this_minute = 0
last_minute = int(time.time() / 60)

def on_request():
    current_minute = int(time.time() / 60)
    if current_minute != last_minute:
        requests_this_minute = 0  # reset
        last_minute = current_minute
    requests_this_minute += 1

The problem: imagine an attacker sends 1000 requests at 14:00:59 and another 1000 at 14:01:01. The per-minute counter resets at 14:01:00, so each minute only shows 1000 requests. But in reality, 2000 requests arrived in a 2-second window — a massive spike that the counter completely misses.

The right way: a deque of timestamps

A sliding window stores the exact timestamp of every request in a collections.deque. When you want the current rate, you evict old entries and count what remains.

from collections import deque
import time

# One deque per IP, one global
ip_window = deque()    # stores timestamps of requests from this IP
WINDOW = 60            # seconds

def on_request(ip: str, now: float):
    # Step 1: Add this request's timestamp
    ip_window.append(now)

    # Step 2: Evict entries older than 60 seconds from the LEFT
    cutoff = now - WINDOW
    while ip_window and ip_window[0] < cutoff:
        ip_window.popleft()   # O(1) — this is why we use deque, not list

    # Step 3: Rate = requests still in window / window size
    rate = len(ip_window) / WINDOW
    return rate  # requests per second over the last 60 seconds

Let's trace through a real example:

Time 14:00:00  → request arrives → deque: [14:00:00]             rate = 1/60 = 0.017 req/s
Time 14:00:01  → request arrives → deque: [14:00:00, 14:00:01]   rate = 2/60 = 0.033 req/s
...normal traffic...
Time 14:01:30  → attacker starts sending 100 req/s
Time 14:01:31  → deque has 100 new entries + a few old ones
               → evict everything before 14:00:31
               → 100 entries remain → rate = 100/60 = 1.67 req/s  ← anomaly detected

The window "slides" forward with real time. Old entries fall off the left; new entries pile up on the right. The rate you get is always "how many requests from this IP in the last 60 seconds."

Why deque and not list?

list.pop(0) is O(n) — it has to shift every element one position to the left every time you evict. With thousands of entries under a DDoS, this becomes very slow.

deque.popleft() is O(1) — it just moves a pointer. No matter how many entries are in the deque, eviction is instant.

Here's the actual implementation from our detector.py:

from collections import deque, defaultdict
from dataclasses import dataclass, field

@dataclass
class IPState:
    """All per-IP sliding window state."""
    timestamps: deque = field(default_factory=deque)  # request timestamps
    error_ts: deque = field(default_factory=deque)     # 4xx/5xx timestamps

# One IPState per IP address, created on first request
ip_states = defaultdict(IPState)

# One global window for all traffic combined
global_ts: deque = deque()

def _evict(dq: deque, window: float, now: float):
    """Remove timestamps older than `window` seconds from the left."""
    cutoff = now - window
    while dq and dq[0] < cutoff:
        dq.popleft()

def _rate(dq: deque, window: float) -> float:
    """Requests per second over the window."""
    return len(dq) / window

def process(entry: dict):
    ip = entry["source_ip"]
    now = entry["_parsed_at"]

    state = ip_states[ip]

    # Update per-IP window
    state.timestamps.append(now)
    _evict(state.timestamps, 60, now)
    ip_rate = _rate(state.timestamps, 60)

    # Update global window
    global_ts.append(now)
    _evict(global_ts, 60, now)
    global_rate = _rate(global_ts, 60)

We maintain two separate windows: one per IP (so we can detect individual attackers) and one global (so we can detect distributed attacks from many IPs). Both are updated on every single request.

Part 3: How the Baseline Learns from Traffic

Detecting anomalies requires knowing what normal looks like. If normal traffic is 50 req/s, then 100 req/s is suspicious. But if normal traffic is 5 req/s, then even 15 req/s might be an attack.

The key insight: normal changes over time. A file-sharing platform sees more traffic during business hours than at 3am. A global service sees more traffic from Asia during Asian business hours. You cannot hardcode a threshold — you have to learn it.

Per-second bucketing

Our baseline engine counts requests per second and stores those counts in a rolling 30-minute window:

from collections import deque
import math
import time
from datetime import datetime

# Stores one count per second, up to 30 minutes worth (1800 buckets)
second_buckets = deque(maxlen=1800)

current_second = int(time.time())
current_count = 0.0

def record(entry: dict):
    global current_second, current_count

    now = entry["_parsed_at"]
    bucket_sec = int(now)

    # When the clock second changes, flush the completed bucket
    if bucket_sec > current_second:
        # Fill any empty seconds (gaps where no traffic arrived)
        for s in range(current_second, bucket_sec):
            second_buckets.append(current_count)
            current_count = 0.0
        current_second = bucket_sec

    current_count += 1.0

So if 47 requests arrived in the second ending at 14:05:00, the deque gets 47.0 appended. If the next second was quiet (zero requests), it gets 0.0. The deque holds up to 1800 of these values — 30 minutes of history.

Computing mean and standard deviation

Every 60 seconds, we recompute the baseline from whatever is in the bucket deque:

def _recalculate():
    samples = list(second_buckets)
    n = len(samples)

    if n < 30:
        return  # not enough data yet, keep the floor values

    # Mean: average requests per second over the last 30 minutes
    mean = sum(samples) / n

    # Standard deviation: how much does traffic vary?
    variance = sum((x - mean) ** 2 for x in samples) / (n - 1)
    stddev = math.sqrt(variance)

    # Apply floor values to prevent false positives during quiet periods
    mean = max(mean, 1.0)      # never let mean drop below 1 req/s
    stddev = max(stddev, 0.5)  # never let stddev drop below 0.5

    effective_mean = mean
    effective_stddev = stddev

Why do we need floor values?

Imagine the server is quiet at 3am with 0.1 req/s average. Without floor values:

mean = 0.1, stddev = 0.05
A burst to just 0.25 req/s gives z-score = (0.25 - 0.1) / 0.05 = 3.0 → false alarm!

With floor_mean = 1.0:

The denominator stays reasonable
You need a genuine spike to trigger detection

Per-hour preference

Traffic patterns repeat by hour of day. We also maintain 24 separate per-hour slot deques:

hour_slots = {h: deque(maxlen=1800) for h in range(24)}

# When flushing a bucket, also file it into the current hour's slot
hour = datetime.fromtimestamp(current_second).hour
hour_slots[hour].append(current_count)

At recalculation time, if the current hour's slot has at least 30 samples, we use it instead of the 30-minute window. This means 9am traffic is compared to yesterday's 9am baseline, not to the 8:30am baseline — much more accurate.

Part 4: How the Detection Logic Makes a Decision

We have a rate (from the sliding window) and a baseline (mean and stddev). How do we decide if the rate is an attack?

Z-score: how many standard deviations above normal?

The z-score is a way of asking "how unusual is this number compared to what we normally see?"

z-score = (current_rate - baseline_mean) / baseline_stddev

If the baseline is mean=5 req/s, stddev=2:

Current rate 7 req/s → z = (7-5)/2 = 1.0 → normal variation
Current rate 11 req/s → z = (11-5)/2 = 3.0 → suspicious
Current rate 25 req/s → z = (25-5)/2 = 10.0 → attack

We flag anything with z-score > 3.0. In a normal distribution, only 0.13% of values naturally exceed z=3. So a false positive rate of 0.13% is acceptable.

Multiplier: absolute rate check

Z-score alone has a weakness: if traffic is very consistent (low stddev), even a modest real attack might not produce a high z-score. So we add a second condition:

if rate > 5 × mean → flag it

If normal is 5 req/s and someone hits 26 req/s, that's more than 5× — flag it regardless of stddev.

Either condition firing triggers the response. Here's the actual detection code:

def _zscore(rate: float, mean: float, stddev: float) -> float:
    if stddev == 0:
        return 0.0
    return (rate - mean) / stddev

def check_anomaly(ip_rate, mean, stddev):
    z_thresh = 3.0   # loaded from config.yaml
    m_thresh = 5.0   # loaded from config.yaml

    ip_z = _zscore(ip_rate, mean, stddev)

    # Either condition triggers the alert
    is_anomaly = (ip_z > z_thresh) or (ip_rate > m_thresh * mean and mean > 0)
    return is_anomaly, ip_z

Error surge tightening

There's a third detection mode for slow attacks. An attacker doing credential stuffing (trying username/password combinations) won't necessarily send huge traffic volumes — but they'll generate lots of 401 Unauthorized responses.

We track error rates separately:

# If this IP's 4xx/5xx rate is 3× the baseline error rate,
# tighten the thresholds for this IP
if err_rate >= 3.0 * baseline_error_mean:
    z_thresh = 2.0   # lower bar → easier to trigger
    m_thresh = 3.0   # lower bar → easier to trigger
    state.tightened = True

This catches slow, stealthy attacks that would otherwise fly under the radar.

Putting it all together

Here's the full process() function that runs on every log line:

def process(self, entry: dict):
    ip = entry["source_ip"]
    now = entry["_parsed_at"]
    status = entry.get("status", 200)

    mean, stddev = self._baseline.get_baseline()

    with self._lock:
        state = self._ip_states[ip]

        # 1. Update sliding windows
        state.timestamps.append(now)
        self._evict(state.timestamps, 60, now)

        self._global_ts.append(now)
        self._evict(self._global_ts, 60, now)

        ip_rate = len(state.timestamps) / 60
        global_rate = len(self._global_ts) / 60

        # 2. Check error surge → tighten thresholds if needed
        if status >= 400:
            state.error_ts.append(now)
        self._evict(state.error_ts, 60, now)
        err_rate = len(state.error_ts) / 60

        z_thresh = 2.0 if state.tightened else 3.0
        m_thresh = 3.0 if state.tightened else 5.0

        # 3. Score and decide
        ip_z = (ip_rate - mean) / stddev if stddev > 0 else 0
        ip_anomaly = (ip_z > z_thresh) or (ip_rate > m_thresh * mean)

        # 4. Fire callback if anomalous (with 10-second cooldown)
        if ip_anomaly:
            last = self._last_fired.get(ip, 0)
            if now - last > 10:
                self._last_fired[ip] = now
                self.on_ip_anomaly(ip, {"ip_rate": ip_rate, "zscore": ip_z, ...})

        # 5. Check global anomaly separately
        g_z = (global_rate - mean) / stddev if stddev > 0 else 0
        if g_z > 3.0 or global_rate > 5.0 * mean:
            if now - self._global_last_fired > 30:
                self._global_last_fired = now
                self.on_global_anomaly({"global_rate": global_rate, ...})

When on_ip_anomaly fires, it calls the blocker. When on_global_anomaly fires, it calls the notifier directly (no ban — you can't ban the whole internet).

Part 5: How iptables Blocks an IP

iptables is the Linux kernel's firewall. It processes every network packet that enters or leaves your machine and applies rules to decide what to do with each one.

Think of it like a bouncer at a club. The bouncer has a list of rules:

Allow anyone with a VIP pass (ACCEPT)
Drop anyone on the blacklist (DROP)
Default: let everyone in (ACCEPT)

When we detect an attack from IP 1.2.3.4, we add a rule to the top of the INPUT chain (the chain that handles incoming packets):

iptables -I INPUT -s 1.2.3.4 -j DROP

Breaking this down:

-I INPUT — Insert at the top of the INPUT chain (before other rules)
-s 1.2.3.4 — Match packets with source IP 1.2.3.4
-j DROP — Silently drop matching packets (no response sent to attacker)

The -I (insert) rather than -A (append) is important. Rules are checked top-to-bottom. By inserting at the top, our DROP rule is checked before any ACCEPT rules — the attacker is blocked immediately.

You can verify the rule was added:

$ iptables -L INPUT -n -v
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  1.2.3.4             0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0           0.0.0.0/0  state RELATED,ESTABLISHED
...

Here's the Python code that issues this command:

import subprocess

def _iptables_add(ip: str):
    """Add a DROP rule for this IP. Check first to avoid duplicates."""
    # Check if rule already exists
    check = subprocess.run(
        ["iptables", "-C", "INPUT", "-s", ip, "-j", "DROP"],
        capture_output=True
    )
    if check.returncode == 0:
        return  # already banned, nothing to do

    # Add the rule
    subprocess.run(
        ["iptables", "-I", "INPUT", "-s", ip, "-j", "DROP"],
        check=True,
        capture_output=True
    )

def _iptables_remove(ip: str):
    """Remove the DROP rule. Loop in case it was added more than once."""
    while True:
        result = subprocess.run(
            ["iptables", "-D", "INPUT", "-s", ip, "-j", "DROP"],
            capture_output=True
        )
        if result.returncode != 0:
            break  # no more rules for this IP

The backoff schedule

We don't ban forever on the first offence. The ban duration follows a backoff schedule:

Offence	Duration
1st ban	10 minutes
2nd ban	30 minutes
3rd ban	2 hours
4th+ ban	Permanent

A background thread (the unbanner) wakes up every 30 seconds and checks whether any bans have expired:

import threading
import time

class UnbannerThread(threading.Thread):
    def run(self):
        while not self._stop.wait(30):  # check every 30 seconds
            expired = self._blocker.get_expired_bans()
            for ip in expired:
                record = self._blocker.unban(ip)
                if record:
                    self._notifier.send_unban(record)  # Slack alert

def get_expired_bans(self) -> list[str]:
    now = datetime.utcnow()
    return [
        ip for ip, rec in self._bans.items()
        if rec.expires_at is not None and now >= rec.expires_at
    ]

When an IP is unbanned, iptables -D INPUT -s <ip> -j DROP removes the rule, and a Slack notification goes out so the team knows the ban was lifted.

Part 6: Slack Alerts

Every ban, unban, and global anomaly sends a Slack message. We use Slack's Incoming Webhooks — a URL you POST JSON to, and it appears as a message in your channel.

import requests
import json

def send_ban(ip: str, info: dict, record):
    text = (
        f":rotating_light: *IP BANNED* — `{ip}`\n"
        f"*Rate:* `{info['ip_rate']:.2f} req/s`  "
        f"*Baseline:* `{info['mean']:.2f}`  "
        f"*Z-score:* `{info['zscore']:.2f}`\n"
        f"*Duration:* `{record.duration_label}`  (ban #{record.ban_count})\n"
        f"*Time:* {timestamp}"
    )
    requests.post(
        webhook_url,
        data=json.dumps({"text": text}),
        headers={"Content-Type": "application/json"},
        timeout=5
    )

Critically, Slack calls happen in a thread pool — they never block the main detection loop. If Slack is slow or down, we don't miss detecting the next attack:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=4)

def send_ban_async(ip, info, record):
    executor.submit(_post_to_slack, ip, info, record)

Part 7: The Live Dashboard

The dashboard is a Flask web app that serves a single HTML page which polls /api/metrics every 3 seconds via JavaScript fetch().

from flask import Flask, jsonify
import psutil

app = Flask(__name__)

@app.route("/api/metrics")
def metrics():
    snap = detector.get_snapshot()
    mean, stddev = baseline.get_baseline()
    banned = blocker.get_banned_ips()

    return jsonify({
        "uptime": get_uptime_string(),
        "global_rate": snap["global_rate"],
        "top_ips": snap["top_ips"],
        "mean": round(mean, 3),
        "stddev": round(stddev, 3),
        "banned": [{"ip": r.ip, "duration": r.duration_label, ...} for r in banned],
        "cpu_pct": psutil.cpu_percent(),
        "mem_pct": psutil.virtual_memory().percent,
        "audit_tail": read_last_20_audit_lines(),
    })

The frontend JavaScript refreshes this every 3 seconds and updates the DOM without a full page reload:

async function refresh() {
    const r = await fetch('/api/metrics');
    const d = await r.json();

    document.getElementById('global-rate').textContent = d.global_rate.toFixed(2) + ' req/s';
    document.getElementById('mean').textContent = d.mean;

    // Color the rate red if it's approaching the anomaly threshold
    const rateEl = document.getElementById('global-rate');
    rateEl.style.color = d.global_rate > d.mean * 3 ? '#f87171' : '#4ade80';

    // Render banned IPs table
    const bannedRows = d.banned.map(b =>
        `<tr><td>${b.ip}</td><td>${b.duration}</td><td>${b.banned_at}</td></tr>`
    ).join('');
    document.getElementById('banned-table').innerHTML = bannedRows || '<tr><td colspan="3">No active bans ✓</td></tr>';
}

setInterval(refresh, 3000);
refresh(); // immediate first load

Part 8: The Audit Log

Every significant event is written to a structured audit log in a format designed to be both human-readable and grep-able:

[2026-04-26T14:08:21Z] BAN ip=1.2.3.4 | condition=zscore=4.21 rate=87.3req/s mean=5.2 | rate=87.300 | baseline=5.200 | duration=10m
[2026-04-26T14:18:22Z] UNBAN ip=1.2.3.4 | condition=zscore=4.21 rate=87.3req/s mean=5.2 | rate=87.300 | baseline=5.200 | duration=10m
[2026-04-26T14:07:06Z] BASELINE_RECALC ip=N/A | condition=recalculation | rate=5.200 | baseline=1.100 | duration=N/A

You can search these logs:

# All bans today
grep "BAN " /var/log/detector/audit.log | grep "2026-04-26"

# All recalculations (to see baseline evolution)
grep "BASELINE_RECALC" /var/log/detector/audit.log

# How often was 1.2.3.4 banned?
grep "ip=1.2.3.4" /var/log/detector/audit.log | grep "^.*BAN "

Putting It All Together

Here's main.py — the entry point that wires every component together:

import queue
import signal
import threading

# 1. Build all components
log_queue = queue.Queue(maxsize=100_000)
monitor = LogMonitor(log_path, log_queue)
baseline = BaselineEngine(cfg)
blocker = BlockerEngine(cfg, audit_path)
notifier = NotifierEngine(cfg)
detector = DetectorEngine(cfg, baseline)

# 2. Wire the anomaly callbacks
def on_ip_anomaly(ip, info):
    if blocker.is_banned(ip):
        return  # already banned, skip
    record = blocker.ban(ip, condition=..., rate=info["ip_rate"], baseline=info["mean"])
    notifier.send_ban(ip, info, record)

def on_global_anomaly(info):
    notifier.send_global_anomaly(info)

detector.on_ip_anomaly = on_ip_anomaly
detector.on_global_anomaly = on_global_anomaly

# 3. Start background threads
monitor.start()
UnbannerThread(blocker, notifier).start()
run_dashboard(host="0.0.0.0", port=8080)

# 4. Main loop — drain queue, feed baseline and detector
while True:
    try:
        entry = log_queue.get(timeout=0.05)
        baseline.record(entry)   # update statistics
        detector.process(entry)  # check for anomalies
    except queue.Empty:
        pass  # no new log lines, loop again

The design is intentionally simple: one hot path (queue drain → baseline → detector) runs in the main thread. All I/O — Slack posts, iptables calls, dashboard requests — happens in separate threads and never blocks detection.

What I Learned

Building this from scratch taught me things that using Fail2Ban never would have:

Deques are the right tool for sliding windows. The O(1) popleft makes a real difference when you're processing thousands of events per second. A list would have worked fine until the DDoS actually started — which is exactly when you need it to be fast.

Baselines must have floors. My first version had no floor values. It triggered constantly at night because the stddev got so low that normal morning traffic looked like an anomaly. One line of code (mean = max(mean, 1.0)) fixed it.

Per-hour slots matter more than you'd think. Without them, the 30-minute window picks up the tail of the previous hour's traffic and skews the baseline for the current hour. The hour slots give you a cleaner "what does this hour normally look like" answer.

iptables rules apply to the host kernel, not the container. The detector container has to run with network_mode: host and CAP_NET_ADMIN. If you run it in a bridge network and add iptables rules inside the container, those rules only affect traffic inside the container's network namespace — not the actual packets arriving at the server.

Thread safety is non-negotiable. The log monitor, the main loop, the Flask dashboard, the unbanner, and the Slack notifier all touch shared state. A threading.Lock() around every mutation took 20 minutes to add and saved hours of debugging mysterious state corruption.

Running It Yourself

The full source code is on GitHub: github.com/Airfluke/hng-detector

To run it on a fresh Ubuntu VPS:

# Install Docker
curl -fsSL https://get.docker.com | sh

# Clone and configure
git clone https://github.com/AirFluke/hng-detector.git
cd hng-detector
cp .env.example .env
nano .env  # add your Slack webhook URL and server IP

# Launch everything
docker compose up -d --build

# Watch the detector work
docker logs -f hng-detector

# Test it with a traffic burst
apt install apache2-utils
ab -n 2000 -c 150 http://localhost/

# Check iptables for the ban
iptables -L INPUT -n -v

The dashboard will be live at http://your-server-ip:8080.

Conclusion

Security tooling doesn't have to be a black box. The core ideas here — a deque-based sliding window, a rolling statistical baseline, z-score anomaly detection, and iptables for enforcement — are each individually simple. The power comes from wiring them together into a continuously running daemon that never sleeps.

If you're new to security tooling, I hope this post demystified what's happening under the hood of the tools you've always treated as magic. Go build something.

Found this useful? Share it with someone learning DevSecOps. Questions? Drop them in the comments.

DEV Community: Abraham Acha

Building a Production-Grade Observability Platform with the LGTM Stack, DORA Metrics & SLOs

Introduction

Why LGTM Over Managed Alternatives?

Architecture Overview

Part 1: Deploying the Full LGTM Stack as Systemd Services

Why systemd over Docker?

One-command deployment

Systemd unit files — the core of the deployment

Checking service status

File layout on the server

Retention periods

Infrastructure as Code — non-negotiable

Part 2: The Four Golden Signals as SLIs

Why Four Golden Signals beat CPU/RAM monitoring

Signal 1 — Latency

Signal 2 — Traffic

Signal 3 — Errors

Signal 4 — Saturation

Part 3: SLOs and Error Budgets

The philosophy

Our SLO targets

Recording rules — pre-computing SLIs

Error Budget Policy

Part 4: DORA Metrics and CI/CD Observability

Why DORA metrics connect to business outcomes

DORA benchmarks

GitHub Actions pushing DORA metrics to Pushgateway

Toil identified and automated

Part 5: Five Grafana Dashboards — All Provisioned as Code

Datasource provisioning with trace correlation

Dashboard 1 — Node Exporter

Dashboard 2 — Blackbox Exporter

Dashboard 3 — DORA Metrics

Dashboard 4 — SLO & Error Budget

Dashboard 5 — Unified Observability (the most important)

📸 [Screenshot: Loki logs panel with clickable trace ID]

Part 6: The Alerting System

All alert rules are version-controlled

Infrastructure alerts

Burn rate alerting — how it reduces alert fatigue

Alertmanager routing and inhibition

Structured Slack template

Part 7: Runbooks and Incident Management

A runbook for every alert

Blameless Post-Incident Review

Part 8: Game Day Results

Scenario 1 — Deployment Failure

Scenario 2 — Latency Injection

Scenario 3 — Resource Pressure

Key Lessons Learned

Conclusion

Building a Production-Grade Observability Platform with LGTM Stack, DORA Metrics & SLOs

Introduction

Why LGTM Over Managed Alternatives?

Architecture Overview

Part 1: Deploying the Full LGTM Stack

Docker Compose — the complete stack

One command to bring everything up

Infrastructure as Code — non-negotiable

Prometheus scrape configuration

Part 2: The Four Golden Signals as SLIs

Why Four Golden Signals beat CPU/RAM monitoring

Signal 1 — Latency

Signal 2 — Traffic

Signal 3 — Errors

Signal 4 — Saturation

Part 3: SLOs and Error Budgets

The philosophy

Our SLO targets

Recording rules for SLIs

Error Budget Policy

Part 4: DORA Metrics and CI/CD Observability

Why DORA metrics connect to business outcomes

DORA benchmarks

GitHub Actions pushing DORA metrics to Pushgateway

DORA recording rules in Prometheus

Toil identified and automated

Part 5: Five Grafana Dashboards — All Provisioned as Code

Grafana provisioning configuration

`init` — generates everything from the manifest

`validate` — five pre-flight checks

`deploy` — brings up the stack and blocks until healthy

`promote` — rolling restart with zero nginx downtime