DEV Community: Adeolu

Building a Production-Grade Observability Platform for the Anvila API with LGTM, SLOs, DORA Metrics, and Game Day Testing

Adeolu — Tue, 19 May 2026 00:08:05 +0000

Introduction

For the HNG DevOps Stage 6 task, our team built a production-grade observability and reliability platform for the Anvila API.

The goal was not just to check whether a server was up or down. We needed to build a monitoring system that could help a team understand the health of an application from different angles:

Is the application available?
Is it fast enough for users?
Are requests failing?
Is the server under pressure?
Are deployments healthy?
Can engineers investigate logs, metrics, and traces from one place?
Can alerts reach the team with enough context to act quickly?

To achieve this, we used the LGTM observability stack:

Loki for logs
Grafana for dashboards
Tempo for traces
Prometheus for metrics

We also added Alertmanager for alert routing, Node Exporter for server metrics, Blackbox Exporter for uptime probing, OpenTelemetry Collector for logs and traces, and a custom GitHub Actions DORA exporter.

Prometheus targets all up

Architecture Overview

Our architecture uses a separate monitoring server. The Anvila API continues to run on its application server, while the monitoring stack runs on a dedicated AWS EC2 instance.

The monitoring server collects:

system metrics from Node Exporter
uptime and response time metrics from Blackbox Exporter
application metrics from the FastAPI /metrics endpoint
logs from the app server through OpenTelemetry Collector and Loki
traces from the instrumented FastAPI staging service through OpenTelemetry and Tempo
CI/CD metrics from GitHub Actions through a custom DORA exporter

Grafana connects all these data sources together so that we can view metrics, logs, and traces in one place.

LGTM dashboards list

At a high level, the data flow is:

Anvila API -> OpenTelemetry Collector -> Loki / Tempo
Anvila API /metrics -> Prometheus
Node Exporter -> Prometheus
Blackbox Exporter -> Prometheus
GitHub Actions exporter -> Prometheus
Prometheus / Loki / Tempo -> Grafana
Prometheus alerts -> Alertmanager -> Slack

Why We Chose LGTM Over Managed Alternatives

Managed observability platforms are useful because they reduce operational work. However, for this task, we chose LGTM because we needed to understand and control the full observability pipeline.

LGTM gave us:

full control over metrics, logs, and traces
version-controlled configuration
no dependency on a paid managed observability provider
a better learning experience for how observability systems actually work
the ability to provision dashboards, alert rules, and data sources as code

Prometheus is strong for metrics and alerting. Loki is useful for storing logs in a way that works well with Grafana. Tempo stores distributed traces and helps us understand request flow. Grafana brings all three together in one interface.

This made LGTM a good fit for a platform engineering task where the goal was to build the stack from the ground up.

Infrastructure as Code and Systemd Deployment

One important requirement was that the stack should be reproducible. We used Terraform to provision the monitoring EC2 server and upload the configuration files.

The services are installed and managed with systemd instead of Docker. This was important because the task update said we should not install Docker on the server.

The stack includes systemd services for:

Prometheus
Grafana
Loki
Tempo
Alertmanager
Node Exporter
Blackbox Exporter
OpenTelemetry Collector
DORA exporter

The repository contains:

Terraform files
Prometheus configuration
Alertmanager configuration
Grafana dashboard JSON files
Loki and Tempo configuration
OpenTelemetry Collector configuration
alert rules
runbooks
SLI/SLO documentation
Game Day documentation

Alert rules YAML

Metrics Collection

Metrics are numerical measurements that tell us what is happening in the system.

We collect several types of metrics:

Server Metrics

Node Exporter gives us system-level metrics such as:

CPU usage
memory usage
disk usage
disk I/O
network I/O
system load

These metrics help us understand saturation, which means how full or stressed the system is.

Node Exporter dashboard

Uptime and HTTP Probe Metrics

Blackbox Exporter probes both the staging and production API URLs.

It checks:

whether the endpoint is reachable
whether it returns a successful HTTP response
how long the response takes
SSL certificate expiry

This is important because it measures the service from the outside, closer to how a user or client would experience it.

Blackbox dashboard

Application Metrics

We also added app-level Prometheus metrics to the FastAPI staging application through a /metrics endpoint.

This gives Prometheus real application request metrics such as:

http_requests_total
http_request_duration_seconds
http_requests_in_progress

These metrics help us track request volume, request latency, and request status codes from inside the application itself.

App metrics target up

App request metrics

App latency histogram

Logs and Traces

Metrics tell us that something is wrong. Logs and traces help us understand why.

We used OpenTelemetry Collector to collect application logs and send them to Loki. In Grafana, we can search these logs using Loki queries.

Loki logs

We also instrumented the staging FastAPI service with OpenTelemetry so that it sends traces to Tempo.

A trace shows the path of a request through the application. This is very useful when investigating slow requests, because it helps identify which endpoint or operation caused the delay.

Tempo traces

The Unified Observability dashboard combines metrics, logs, and traces. The idea is that if latency or error rate increases, an engineer can look at the metric, check related logs in Loki, and then inspect traces in Tempo.

Unified observability dashboard

The Four Golden Signals

The Four Golden Signals are a simple way to define service reliability from a user-focused perspective.

They are:

Latency
Traffic
Errors
Saturation

Latency

Latency means how long it takes the service to respond.

For Anvila, we track latency using both Blackbox response time and application request duration metrics.

Example PromQL:

histogram_quantile(
  0.95,
  sum by (le, path) (
    rate(http_request_duration_seconds_bucket{job="anvila-api-staging",status!~"5.."}[5m])
  )
)

This tells us the 95th percentile request latency for successful requests.

Traffic

Traffic means how much demand the service is receiving.

Example PromQL:

sum(rate(http_requests_total{job="anvila-api-staging"}[5m]))

This tells us the number of requests per second.

Errors

Errors measure how many requests are failing.

Example PromQL:

sum(rate(http_requests_total{job="anvila-api-staging",status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_requests_total{job="anvila-api-staging"}[5m])), 1)

This gives us the ratio of failed requests.

Saturation

Saturation shows how full the system is.

For example, CPU usage:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

This goes beyond simply checking if the server is alive. A server can be up but overloaded. The Four Golden Signals help us understand the actual quality of the service.

SLOs and Error Budgets

An SLI is a Service Level Indicator. It is a measurement.

An SLO is a Service Level Objective. It is the target we want to meet.

An error budget is the amount of unreliability we are allowed within a time window.

For Anvila, our main availability SLO is:

99.5% successful probes over 30 days

The error budget calculation is:

(1 - 0.995) * 30 days = 0.15 days = 3.6 hours

This means the service can be unavailable for about 3 hours and 36 minutes in a 30-day window before the error budget is fully consumed.

The SLO dashboard shows:

30-day availability SLI
SLO target
error budget remaining
error budget time remaining
burn rate
7-day and 30-day compliance history

SLO and Error Budget dashboard

Burn Rate Alerting

Burn rate tells us how quickly the service is consuming its error budget.

This is better than simple alerting because it reduces alert fatigue.

For example, a short small failure may not need to wake everyone up. But if the service is burning through its error budget very quickly, the team should respond immediately.

We configured:

Fast Burn alert: critical
Slow Burn alert: warning

The Fast Burn alert means the system is consuming the error budget too quickly and needs immediate action.

The Slow Burn alert means the system may become a serious incident if the trend continues.

DORA Metrics

DORA metrics help connect engineering work to business outcomes.

The four DORA metrics are:

Deployment Frequency
Lead Time for Changes
Change Failure Rate
Mean Time to Restore

These metrics help answer questions such as:

How often do we deploy?
How long does it take for code to reach production?
How often do deployments fail?
How quickly can we recover from incidents?

We built a GitHub Actions DORA exporter that exposes deployment workflow data to Prometheus.

The dashboard shows:

deployment frequency
lead time
change failure rate
MTTR
deployment frequency classification

DORA dashboard

Our current DORA implementation measures commit-to-workflow-completion lead time. A future improvement would be to break it into more detailed sub-intervals such as commit time, workflow trigger time, workflow completion time, and deployment confirmation time.

Alerting and Slack Notifications

Alert rules are stored in version-controlled YAML files, not manually created in the Grafana UI.

We configured alerts for:

host downtime
high CPU
high memory
disk usage
SLO burn rate
change failure rate
MTTR threshold

Alertmanager handles routing and sends alerts to Slack.

The Alertmanager configuration includes:

route grouping by alert name, service, and severity
warning and critical routes
inhibition rules
Slack receiver configuration

Alertmanager config

We also used a structured Slack template. The alert payload includes:

alert name
severity
affected host or target
current value
dashboard link
runbook link
firing or resolved status

Slack template

Here is an example of a firing and resolved Slack notification:

Slack firing and resolved alert

Runbooks and Incident Management

Every alert needs a runbook. A runbook explains what the alert means and what to do first.

Our runbooks include:

what the alert means
likely causes
first investigation steps
how to resolve it
when to roll back
when to escalate

Example runbook:

[SLO burn rate runbook]

We also documented a blameless post-incident review for the simulated latency incident.

The goal of a blameless PIR is not to blame a person. The goal is to understand what happened, what the impact was, what went well, what did not go well, and what should be improved.

Game Day Testing

Game Day is where we intentionally create controlled failures to test whether our monitoring and response system works.

We ran three scenarios.

Scenario 1: Deployment Failure

We created a temporary PR with an intentionally failing GitHub Actions workflow.

The goal was to prove that a deployment or pipeline failure can be detected without touching production.

The PR was closed without merging after screenshots were captured.

Game Day deployment failure PR

Game Day deployment failure check

Scenario 2: Latency Injection

During the latency Game Day, we temporarily added a 2-second delay to the staging root endpoint and generated test requests. The evidence below shows the resulting Tempo traces and the recovery after the delay was removed.

Game Day latency traces

Game Day latency recovery

Scenario 3: Resource Pressure

We created CPU pressure on the monitoring server using controlled background processes.

This triggered a CPU warning alert in Slack. After stopping the pressure, we confirmed the resolved alert was also sent.

Game Day resource pressure trigger

Game Day resource pressure Slack alert

Game Day resource pressure recovery

Toil Identified and Automation

Toil is repetitive manual work that can be automated.

We identified several examples of toil:

1. Manual Dashboard Creation

Creating dashboards manually in Grafana is slow and hard to reproduce.

Automation: All dashboards are stored as JSON and provisioned automatically.

2. Manual Monitoring Server Setup

Installing each service manually would be time-consuming and error prone.

Automation: Terraform creates the monitoring server, and systemd installation scripts configure the observability stack.

3. Vague Alert Investigation

If an alert only says, "CPU high", the engineer still has to search for the server, dashboard, and runbook.

Automation: Alertmanager Slack messages include service, host, metric value, dashboard link, and runbook link.

4. Manual CI/CD Metrics Collection

Checking GitHub Actions manually does not scale.

Automation: The DORA exporter pulls GitHub Actions data and exposes it as Prometheus metrics.

What We Learned

This project helped us understand that observability is more than installing tools.

The tools are only useful when they are connected to reliability goals.

Prometheus helped us collect and alert on metrics. Loki helped us inspect logs. Tempo helped us understand request traces. Grafana helped us bring everything together. Alertmanager helped us route actionable alerts to Slack.

We also learned that good reliability engineering requires:

clear SLIs
realistic SLOs
error budgets
runbooks
incident reviews
controlled failure testing

Current Limitations and Next Steps

The platform is functional, but there are still improvements we can make.

Current limitations:

App-level /metrics is currently verified on staging.
Production is monitored through Blackbox probes and shared host-level metrics.
DORA lead time is currently simplified to commit-to-workflow-completion.
A follow-up PR was opened to normalize unmatched route labels in Prometheus metrics and reduce high-cardinality risk.

Next steps:

promote app-level metrics to production after staging validation
merge the metrics label normalization improvement
add more detailed DORA lead-time sub-intervals
add deployment annotations to Grafana dashboards
continue reviewing SLOs as the service matures

Conclusion

For this task, we built a complete observability platform around the Anvila API using the LGTM stack.

We deployed the monitoring stack with Terraform and systemd, collected metrics, logs, and traces, built dashboards, configured Slack alerts, wrote runbooks, and tested the system through Game Day simulations.

The biggest lesson is that observability is not just about knowing when something is broken. It is about giving the team enough context to understand the problem, respond quickly, and improve the system after every incident.

Repository:

https://github.com/Adeolu1024/anvila-observability-platform

Evidence screenshots:

https://drive.google.com/drive/folders/1v_DiT6XQtq0iJtn5JqxfZecAmxhCdd82?usp=sharing```

Building SwiftDeploy: A Declarative Infrastructure CLI with Observability and Policy Enforcement

Adeolu — Wed, 06 May 2026 20:54:03 +0000

What Is This Project?

SwiftDeploy is a command-line tool that automatically sets up and manages web application deployments. Instead of manually configuring Docker containers, Nginx, and monitoring, you write one file (manifest.yaml) that describes what you want, and the tool builds everything for you.

The project was built in two parts:

Stage 4A: Basic infrastructure generation and container management
Stage 4B: Monitoring, policy enforcement, and audit logging

The Core Idea: Declarative Configuration

In traditional DevOps, you manually write configuration files for each service. With SwiftDeploy, you write a single manifest file, and the tool generates all the configuration files automatically.

manifest.yaml (the only file you edit manually):

services:
  image: swiftdeploy-keeds-api:v1.0.0
  port: 5000
  name: api-service
  mode: stable

nginx:
  image: nginx:alpine
  port: 8080
  proxy_timeout: 30s

network:
  name: swiftdeploy-net
  driver_type: bridge

From this one file, SwiftDeploy generates:

nginx.conf (web server configuration)
docker-compose.yml (container orchestration)
All the settings for monitoring and policy checks

How the Tool Works

The CLI tool (swiftdeploy) has several commands:

Command	What It Does
`init`	Reads manifest.yaml and generates nginx.conf + docker-compose.yml
`validate`	Checks if everything is ready for deployment
`deploy`	Starts all containers and waits for them to be healthy
`promote canary/stable`	Switches between stable and canary modes
`status`	Shows a live dashboard with metrics and policy compliance
`audit`	Generates a report of all events and policy violations
`teardown`	Stops and removes all containers

Stage 4A: The Foundation

API Service

The API service is a Python Flask application that serves:

GET / — Welcome message with current mode and version
GET /healthz — Health check endpoint
POST /chaos — Simulates problems for testing (only in canary mode)

Nginx Proxy

Nginx acts as a reverse proxy, routing all traffic to the API service. It:

Listens on port 8080 (configurable)
Forwards requests to the API service
Returns JSON error responses for 502, 503, 504 errors
Logs all requests in a specific format

Docker Compose

Docker Compose manages all containers:

API service (your application)
Nginx (web server/proxy)
OPA (policy engine, added in Stage 4B)

Stage 4B: Observability and Policy Enforcement

The /metrics Endpoint

The API service now exposes a /metrics endpoint that reports statistics in Prometheus format:

http_requests_total{method="GET",path="/healthz",status_code="200"} 42
http_request_duration_seconds_bucket{le="0.1"} 35
app_uptime_seconds 847
app_mode 0
chaos_active 0

These metrics tell you:

How many requests have been made
How fast responses are
How long the app has been running
Whether you're in stable or canary mode
Whether chaos testing is active

OPA: The Policy Engine

OPA (Open Policy Agent) is a separate container that acts like a security guard. Before you can deploy or promote, the CLI asks OPA: "Is it safe?"

Why use OPA instead of checking directly in the CLI?**

Policies are separate from code — easier to update
If OPA crashes, the CLI still works (just warns you)
OPA is not accessible from the internet (security)

The Two Policies

Infrastructure Policy** (checks before deploy):

Is there enough disk space? (must be > 10GB)
Is the CPU overloaded? (must be < 2.0)

Canary Safety Policy** (checks before promoting to canary):

Is the error rate too high? (must be < 1%)
Is the response time too slow? (P99 must be < 500ms)

Data-Driven Thresholds

The actual numbers (10GB, 2.0, 1%, 500ms) are stored in a separate JSON file, not in the policy code. This means you can change the limits without modifying the policy logic.

thresholds.json:

{
  "infrastructure": {
    "min_disk_gb": 10,
    "max_cpu_load": 2.0
  },
  "canary": {
    "max_error_rate": 0.01,
    "max_p99_latency_ms": 500
  }
}

The Status Dashboard

The swiftdeploy status command shows a live dashboard:

╔═══════════════════════════════════════╗
║     SwiftDeploy Status Dashboard      ║
╠═══════════════════════════════════════╣
║ Mode: canary                         ║
║ Chaos: none                          ║
║ Req/s: 0.98                          ║
║ P99 Latency: 5ms                     ║
║ Error Rate: 0.00%                    ║
║ Uptime: 133s                         ║
╠═══════════════════════════════════════╣
║ Policy Compliance                    ║
║   Infrastructure: PASS               ║
║   Canary Safety:  PASS               ║
╚═══════════════════════════════════════╝

Every time the dashboard refreshes, it saves the data to history.jsonl for the audit trail.

The Audit Report

The swiftdeploy audit command reads history.jsonl and generates audit_report.md with:

A timeline of all events (mode changes, status updates)
A list of policy violations (when checks failed)

Bugs We Fixed

Bug 1: OPA Crashed on Startup

Problem: OPA wouldn't start because of "conflicting rules" error.

Cause: We wrote default deny := [] in the Rego file, which conflicted with deny contains msg if { ... }.

Fix: Removed the default deny := [] line. The contains keyword handles empty sets automatically.

Bug 2: OPA Couldn't Find Threshold Values

Problem: OPA loaded the policy files but couldn't find the threshold numbers.

Cause: The JSON file was in the wrong directory. OPA loads files based on their path structure.

Fix: Moved thresholds.json into a swiftdeploy/ subdirectory so OPA could find it at the correct data path.

Bug 3: Status Dashboard Showed "FAIL" Incorrectly

Problem: The dashboard showed "Infrastructure: FAIL" and "Canary Safety: FAIL" even when everything was within limits.

Cause: The CLI didn't send a timestamp field to OPA. The policy rules need input.timestamp to work. Without it, the rules failed, and the CLI defaulted to "FAIL".

Fix: Added timestamp to all OPA queries.

Bug 4: Nginx Couldn't Find the API Service

Problem: Nginx returned 502 errors saying it couldn't resolve the API service hostname.

Cause: Nginx tried to find the API service at startup, but the container wasn't running yet.

Fix: Added Docker's internal DNS resolver (127.0.0.11) and used a variable for the proxy address. This tells Nginx to look up the hostname when a request comes in, not at startup.

Bug 5: Container Didn't Update After Promoting

Problem: After switching to canary mode, the container was still running in stable mode.

Cause: Using docker compose restart doesn't reload environment variables from the updated docker-compose.yml.

Fix: Changed to docker compose up -d --no-deps <service>, which recreates the container with new settings.

Bug 6: Nginx Permission Denied

Problem: Nginx failed to start with "Permission denied" errors.

Cause: We set user: nginx and removed all Linux capabilities, which prevented Nginx from creating necessary directories.

Fix: Removed the explicit user setting. The official Nginx image handles user switching internally.

Key Design Decisions

Why a Separate OPA Container?

The task required: "The CLI must not make any allow/deny decision itself."

This means:

The CLI asks OPA for permission before every deploy/promote
OPA returns "allowed" or "denied" with a reason
The CLI never makes the decision itself

Benefits:

Policies can be updated without changing the CLI
If OPA is down, the CLI warns but continues
All decisions are logged with reasoning

Why Data-Driven Thresholds?

The task required: "Threshold values must not be hardcoded inside Rego files."

This means the numbers (10GB, 2.0, 1%, 500ms) are in a separate JSON file, not in the policy code. This makes it easy to change limits without touching the policy logic.

Why Separate Policy Files?

The task required: "Organise policies by domain. Each domain owns exactly one question."

This means:

infrastructure.rego only checks disk and CPU
canary.rego only checks error rate and latency
Changing one policy never requires changing another

How the Pieces Fit Together

User runs: swiftdeploy deploy
    │
    ▼
CLI gets host stats (disk, CPU)
    │
    ▼
CLI asks OPA: "Is it safe to deploy?"
    │
    ▼
OPA checks infrastructure policy
    │
    ├── If safe → Start containers
    │
    └── If not safe → Block with reason

User runs: swiftdeploy promote canary
    │
    ▼
CLI scrapes /metrics endpoint
    │
    ▼
CLI calculates error rate and P99 latency
    │
    ▼
CLI asks OPA: "Is it safe to promote?"
    │
    ▼
OPA checks canary safety policy
    │
    ├── If safe → Switch to canary mode
    │
    └── If not safe → Block with reason

Lessons Learned

One file can drive everything**: A single manifest file can generate all the configuration files you need.
Policies should be separate from code**: Using OPA makes policies easier to update and test.
Always handle failures gracefully**: If OPA is down, the CLI warns but continues working.
Simple templates work fine**: You don't need complex template engines for configuration files.
Container recreation vs restart**: Restarting a container doesn't reload environment variables. You need to recreate it.
Docker DNS is important**: Nginx needs to know how to find containers by name, which requires Docker's internal DNS resolver.

Summary

SwiftDeploy is a tool that:

Takes a single manifest file as input
Generates all configuration files automatically
Manages container lifecycle (deploy, promote, teardown)
Enforces safety policies via OPA before deploy/promote
Provides monitoring via /metrics endpoint
Tracks all events in an audit log

The key innovation is that everything is driven by one file, and all safety checks happen automatically before any deployment action.

This article documents the development of SwiftDeploy as part of the HNG Internship DevOps Track, Stage 4.