DEV Community: Adeolu

[Boost]

Adeolu — Wed, 10 Jun 2026 20:11:36 +0000

Adeolu

Jun 10

Engineering Design Document: Reusable Observability Platform V2

#devops #observability #architecture #sre

19 min read

Engineering Design Document: Reusable Observability Platform V2

Adeolu — Wed, 10 Jun 2026 20:11:24 +0000

A production-focused redesign of a Stage 6 LGTM observability platform, moving from a single-service Anvila monitoring setup to a reusable, secure, highly available observability platform.

Executive Summary

This document proposes V2 of the observability platform I built during Stage 6. V1 was validated using the Anvila API as the first monitored service, so many names, dashboards, alerts, and targets were Anvila-specific. However, the real architectural idea was broader than Anvila: build a reusable internal monitoring and reliability platform that could collect metrics, logs, traces, service-level objectives, DORA metrics, and alerts for production applications.

In this document, I treat Anvila as the first customer of the platform, not as the only possible customer. V1 proved that the stack worked for one real API. V2 redesigns it into a highly available, secure, multi-environment observability platform that can onboard many services without rebuilding the whole system each time. "Highly available" means the platform should keep working even if one server or component fails. The main changes are: replacing the single monitoring server with a resilient telemetry architecture, hardening access to observability tools, improving OpenTelemetry collection, making alert routing ownership-aware, storing telemetry with explicit retention and durability policies, and turning SLO and DORA measurements into enforceable release decisions.

The target reader for this document is a Principal Engineer reviewing whether the proposed architecture can survive operational pressure, not whether the dashboards look impressive. For clarity, I still reference Anvila throughout the document because it was the real service used to test V1.

1. V1 Architecture Critique

V1 Overview

V1 was a single-service observability platform deployed on a dedicated AWS EC2 monitoring server. It monitored Anvila as the first real application, so the implementation was branded and configured around Anvila. The application server ran the Anvila API through Nginx and PM2, with staging and production processes exposed on separate local ports. PM2 is a process manager; in this setup, it kept the FastAPI backend running and allowed the team to restart staging or production processes. The monitoring server ran the LGTM stack as systemd services rather than Docker. LGTM means Loki, Grafana, Tempo, and Prometheus:

Prometheus collected metrics, which are numeric measurements such as request count, latency, CPU usage, and error rate.
Loki stored logs, which are timestamped records of what the application and servers are doing.
Tempo stored traces, which show the path and timing of a request across services.
Grafana visualized dashboards by reading data from Prometheus, Loki, and Tempo.
Alertmanager routed alerts to #DevOps-Alerts.
Node Exporter collected host-level CPU, memory, disk, filesystem, and network metrics.
Blackbox Exporter probed the public staging and production API URLs.
OpenTelemetry Collector received application traces and shipped logs.
A custom DORA exporter scraped GitHub Actions workflow data into Prometheus.
A later Tempo recent-traces exporter exposed trace summaries to Grafana through Prometheus.

The backend also received application-level instrumentation. Instrumentation means adding code or configuration so the system can report what is happening inside it. The FastAPI app exposed /metrics using prometheus_client, with counters, histograms, and in-progress gauges:

http_requests_total
http_request_duration_seconds
http_requests_in_progress

The route-labeling logic normalized unmatched paths into a controlled label to avoid Prometheus label-cardinality explosion from random scanner URLs.

Dashboards were provisioned as JSON, and alert rules were committed as YAML. V1 included SLI/SLO definitions, an error budget policy, runbooks, a blameless post-incident review, and Game Day evidence for deployment failure, latency injection, and resource pressure.

Exact Breaking Points

The first major breaking point was the single monitoring EC2 instance. If that instance failed, the team lost Prometheus, Grafana, Loki, Tempo, Alertmanager, dashboards, and local alert routing at the same time. That is unacceptable for production because the observability platform becomes unavailable exactly when operators may need it most. This is the opposite of high availability: one machine became a single point of failure.

V2 fix: split the platform into multiple highly available components. Run at least two OpenTelemetry Collectors behind an internal load balancer, run Grafana in a highly available setup, and move telemetry storage to durable backends. One server failure should reduce capacity, not remove observability.

The second breaking point was local disk dependency. Prometheus, Loki, Tempo, and Grafana state lived on one machine. A disk fill, filesystem issue, or instance replacement could destroy recent telemetry unless backups were handled outside the documented workflow. V1 had retention periods, but retention is not durability.

V2 fix: move logs and traces to object-storage-backed Loki and Tempo, and move metrics to Prometheus remote write or a Prometheus-compatible long-term store such as Mimir. This means telemetry survives instance replacement and is not tied to one local disk.

The third breaking point was access control. Grafana, Prometheus, and Alertmanager were exposed on public ports and protected mainly by a security group allowlist. That is better than open internet access, but it is not strong enough for production. IP allowlists break when team members change networks, and they do not provide identity, auditability, role-based permissions, or revocation at the user level.

V2 fix: put observability tools behind an identity-aware access layer. Grafana should use SSO, MFA, and RBAC, while Prometheus, Loki, Tempo, and Alertmanager APIs should stay private. Access should be based on who the engineer is and what role they have, not only on their IP address.

The fourth breaking point was telemetry ingestion coupling. The application depended on PM2 startup commands and OpenTelemetry runtime instrumentation. If the process was restarted without the instrumentation wrapper, traces could silently disappear while the app continued serving traffic. That creates false confidence: dashboards still exist, but the signal is incomplete.

V2 fix: make telemetry configuration part of the deployment contract. Services should send telemetry to a stable internal OTLP endpoint, and instrumentation settings should be versioned with the service deployment. Health checks should also verify that metrics, logs, and traces are arriving, not only that the API returns HTTP 200.

The fifth breaking point was incomplete DORA accuracy. Deployment frequency and change failure rate were reasonable approximations from GitHub Actions, but lead time and MTTR were not fully event-driven. Deployment confirmation was inferred from workflow success, and MTTR used a placeholder/manual incident metric. This is acceptable for a Stage 6 demo, but it weakens executive reporting because the data can overstate delivery health.

V2 fix: make DORA events explicit. The deployment pipeline should emit deployment started, deployment completed, rollback started, rollback completed, incident opened, and incident resolved events. The DORA exporter should report those real events instead of approximating deployment confirmation and MTTR.

The sixth breaking point was environment and service modeling. Staging and production were monitored, but the stack was not yet designed as a reusable multi-service platform. The labels were Anvila-specific, ownership routing was basic, and onboarding another service would require manual config changes and dashboard duplication. A stronger platform would let another application define its service name, owners, SLOs, alert routes, and dashboards through a repeatable template.

V2 fix: introduce a service registry such as observability_services.yml. Each service defines its name, environment, owners, SLO profile, dashboard folder, and alert route. New services are onboarded by adding a registry entry and using shared dashboard and alert templates.

The seventh breaking point was operational drift. Terraform created the server and uploaded config, but the installation relied heavily on remote shell scripts and mutable systemd services. After deployment, manual server changes could drift away from Git without immediate detection.

V2 fix: reduce mutable server configuration. Terraform should provision infrastructure, while service configuration should move toward immutable images, cloud-init, Ansible, or container orchestration. Config changes should be reviewed in Git and redeployed, not manually patched on the server.

Security Blind Spots

V1 left several security gaps because the immediate goal was proving observability capability under pressure.

Secrets management was too manual. Slack webhook URLs, GitHub tokens, Brevo keys, Terraform variables, and environment-specific credentials were handled through local files or server-side environment files. In a production workflow, files such as private keys, Terraform state, and terraform.tfvars must never live in a publishable repository. They should be stored outside Git, encrypted where possible, rotated if exposure is suspected, and replaced by references to a managed secrets system.

Internal services were not consistently authenticated. Loki had auth_enabled: false, and Prometheus, Alertmanager, and Tempo were designed around network trust rather than service identity. That is common for demos, but production needs stronger boundaries.

Grafana access did not have enterprise-grade identity controls. There was no documented SSO, MFA, per-team RBAC, or audit trail for dashboard and datasource access.

The attack surface was larger than needed. Public access to Grafana, Prometheus, and Alertmanager ports, even through allowlists, increases exposure. Observability systems often contain sensitive data: URLs, headers, traces, stack traces, user IDs, deployment metadata, internal hostnames, and incident timelines.

Log and trace data did not have a documented PII redaction policy. The backend did make good security choices around OAuth logs by hashing emails and avoiding raw token logging, but the wider telemetry platform lacked a formal rule for what must never enter logs, spans, or labels.

2. New Features Fully Designed

Feature 1: Highly Available Telemetry Control Plane

What it does and why it is needed:

V2 replaces the single monitoring server with a highly available observability control plane. A control plane is the part of the system that receives, processes, stores, and routes observability data. The minimum production version runs two OpenTelemetry Collectors behind an internal load balancer, a highly available Grafana instance, and durable backend storage for metrics, logs, and traces. The goal is to ensure that one node failure does not remove visibility during an incident.

Architectural integration:

Application services send OTLP traces, logs, and metrics to an internal telemetry endpoint. OTLP means OpenTelemetry Protocol; it is the standard protocol applications use to send telemetry to OpenTelemetry Collectors. The endpoint load-balances across OpenTelemetry Collectors. Collectors apply batching, memory limits, retries, redaction processors, and routing rules before exporting telemetry to the storage layer.

Prometheus either runs in high-availability pair mode with remote write, or V2 adopts a Prometheus-compatible long-term store such as Grafana Mimir. Remote write means Prometheus keeps scraping metrics but sends a copy to a more durable backend. Loki stores logs using object storage for durability. Tempo stores traces using object storage-backed blocks. Grafana reads from Prometheus/Mimir, Loki, and Tempo.

Data model changes:

Telemetry itself is not stored in the Anvila relational database, but V2 introduces an observability_services.yml registry:

services:
  - service_name: anvila-api
    owner: anvila-devops
    tier: user-facing
    environments: [staging, production]
    slo_profile: public-api-standard
    slack_channel: "#DevOps-Alerts"

This registry becomes the source of truth for labels, alert routes, dashboard folders, and service ownership. If another application joins later, it gets another entry in the same file instead of a separate hand-built monitoring stack.

Trade-offs:

This increases operational complexity and cost. V1 was cheap and simple because everything lived on one EC2 instance. V2 sacrifices simplicity for survivability. The cost is justified because observability must remain available during failures.

Feature 2: Identity-Aware Access Layer

What it does and why it is needed:

V2 puts Grafana, Prometheus, Alertmanager, and trace/log exploration behind an identity-aware access layer. Engineers authenticate through SSO with MFA. Access is granted by team and role, not by IP address.

Architectural integration:

Grafana is placed behind a private ALB or VPN-accessible endpoint. Authentication is delegated to an identity provider. Prometheus, Alertmanager, Loki, and Tempo APIs are not directly public. Engineers query them through Grafana or through short-lived authenticated access paths.

Data model changes:

The observability service registry gains ownership metadata:

owners:
  - team: anvila-devops
    grafana_role: editor
    alert_contact: "#DevOps-Alerts"
    escalation_policy: anvila-primary

Grafana teams map to this registry. Alertmanager routes use the same ownership source.

Trade-offs:

Identity-aware access adds setup time and may slow emergency access if misconfigured. The trade-off is acceptable because production telemetry can contain sensitive operational and user-adjacent data.

Feature 3: Release Health Gate Based on SLO Burn and DORA Signals

What it does and why it is needed:

V1 showed SLO burn and DORA metrics on dashboards, but V2 turns those signals into release policy. An SLO, or Service Level Objective, is a reliability target such as "99.5% of requests should succeed." An error budget is the amount of failure the service can tolerate before it breaks that target. DORA metrics measure software delivery performance: deployment frequency, lead time for changes, change failure rate, and mean time to restore. In V2, deployments can be blocked, delayed, or escalated when the service is burning error budget too quickly or when change failure rate is above threshold.

The important change is that observability stops being only something engineers look at after a problem. It becomes part of the deployment decision. If the platform already knows the service is unhealthy, the release pipeline should not blindly push more change into production. This is the same idea as a safety gate: before the deployment continues, the system checks whether reliability conditions are acceptable.

SLO burn means the service is consuming its error budget. A slow burn means the service is getting worse gradually. A fast burn means the service is failing quickly enough that the team may break the SLO soon. For example, if the availability target is 99.5%, the service only has 0.5% failure allowance for the window. A fast burn alert means that allowance is being consumed too quickly.

Architectural integration:

CI/CD queries a reliability policy endpoint before production deployment. CI/CD means Continuous Integration and Continuous Deployment: the automated path from code change to deployed service. The policy endpoint reads current SLO burn rate, active critical alerts, deployment failure history, and service tier. For Anvila API, production deployment is blocked if:

fast burn is active;
availability budget is fully consumed;
unresolved critical incident exists;
change failure rate exceeds 15% over the configured window.

The policy endpoint does not replace the deployment pipeline. It gives the pipeline a decision: allow, warn, or block. The deployment tool sends context such as service name, environment, commit SHA, actor, and target version. The policy service then checks Prometheus or Mimir for SLO burn, checks Alertmanager for active critical alerts, checks recent deployment history, and returns a decision with reasons.

Example response:

{
  "decision": "block",
  "service": "anvila-api",
  "environment": "production",
  "reasons": [
    "SLOFastBurn is active",
    "Change failure rate is 18%, above the 15% threshold"
  ]
}

This matters because it gives the operator a clear explanation. The system should not just say "deployment blocked." It should explain which reliability rule failed and what evidence caused the block.

Data model changes:

V2 introduces a small reliability metadata database:

CREATE TABLE service_release_policies (
  id UUID PRIMARY KEY,
  service_name TEXT NOT NULL,
  environment TEXT NOT NULL,
  max_change_failure_rate NUMERIC NOT NULL,
  block_on_fast_burn BOOLEAN NOT NULL DEFAULT TRUE,
  block_on_open_critical BOOLEAN NOT NULL DEFAULT TRUE,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE release_decisions (
  id UUID PRIMARY KEY,
  service_name TEXT NOT NULL,
  environment TEXT NOT NULL,
  commit_sha TEXT NOT NULL,
  decision TEXT NOT NULL,
  reasons JSONB NOT NULL,
  requested_by TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The service_release_policies table stores the rules for each service and environment. A user-facing production API can have stricter rules than an internal staging service. For example, production might block deployment during fast burn, while staging might only warn.

The release_decisions table stores every decision made by the policy service. This creates an audit trail. If someone asks why a deployment was blocked or allowed, the team can inspect the commit SHA, environment, decision, reasons, requester, and timestamp.

Trade-offs:

This can slow feature delivery during reliability incidents. That is intentional. V2 chooses controlled delivery over shipping into known instability.

The cost of this feature is extra operational complexity. The team now needs a policy service, a metadata database, and reliable integrations with CI/CD, Prometheus, and Alertmanager. There is also a risk of false positives: a bad alert rule could block a safe deployment. To manage that risk, the policy should support an emergency override path that requires approval, records the reason, and notifies the team.

The benefit is that release decisions become evidence-based. Instead of relying on a human to remember to check dashboards before deployment, the platform checks the most important reliability signals automatically.

Feature 4: Telemetry Data Hygiene and Redaction

What it does and why it is needed:

V2 standardizes what can enter logs, metrics, and traces. It prevents secrets, OAuth codes, access tokens, raw emails, payment IDs, and high-cardinality labels from polluting telemetry. High-cardinality labels are labels with too many possible values, such as raw URLs, user emails, random IDs, or search terms. They can make Prometheus expensive, noisy, and slow.

Architectural integration:

OpenTelemetry Collector processors redact sensitive fields before exporting. Application logging uses structured JSON with approved fields. Metrics labels are restricted to bounded values such as method, route, status, service, and environment.

Data model changes:

An observability_redaction_rules.yml file defines redaction policies:

redact_fields:
  - authorization
  - cookie
  - access_token
  - refresh_token
  - oauth_code
  - github_token
hash_fields:
  - user_email
  - user_id
drop_span_attributes:
  - http.request.header.authorization

Trade-offs:

Redaction can remove useful debugging context. V2 chooses privacy and security over convenience. Debug-only access to sensitive values should come from controlled application debugging, not broad telemetry storage.

3. Production Readiness

Security

Authentication and authorization are treated as separate concerns. Authentication, often shortened to AuthN, proves who the user or service is. Authorization, often shortened to AuthZ, decides what that identity is allowed to do.

Human authentication uses SSO with MFA for Grafana and all human access to observability tools. SSO means Single Sign-On, where users log in through a central identity provider. MFA means Multi-Factor Authentication, where login requires an extra proof beyond a password. Authorization is RBAC-based, meaning Role-Based Access Control: viewers can inspect dashboards, on-call engineers can silence alerts, and platform admins can edit datasources and alert rules. Direct API access to Prometheus, Loki, Tempo, and Alertmanager is blocked from the public internet.

Service-to-service authentication uses private networking plus cloud identity where available. GitHub Actions uses OIDC, or OpenID Connect, to assume AWS roles instead of storing long-lived cloud keys. Secrets move into AWS Secrets Manager or SSM Parameter Store. Terraform receives only secret references, not plaintext secrets. Slack, GitHub, Brevo, Stripe, Gemini, and database credentials are rotated and never stored in Git.

Input validation is enforced at the application and telemetry layers. The FastAPI app already uses typed Pydantic settings and route-level models. V2 extends this by validating telemetry labels and dropping unknown high-cardinality labels at the collector.

The attack surface is minimized by private networking. Only Grafana is reachable through an authenticated access layer. OTLP ports are internal. Node Exporter is reachable only from the collector or Prometheus security group. Admin APIs are private.

Scalability

Horizontal scaling boundaries are explicit:

Anvila API scales horizontally behind a load balancer.
OpenTelemetry Collectors scale horizontally and are stateless.
Grafana scales horizontally behind a load balancer with external database/session storage.
Logs and traces scale through object-storage-backed Loki and Tempo.
Metrics scale through Prometheus HA with remote write or a Mimir-compatible backend.
Celery workers scale independently for persona generation. Celery is a Python background job system. It lets slow work, such as LLM calls or GitHub publishing, run outside the user-facing HTTP request.

Caching should be precise, not generic. Redis is used for session-adjacent ephemeral state, Celery broker/result coordination, persona generation events, and short-lived rate limit counters. Redis is an in-memory data store, which makes it fast for temporary data. Cache keys must have TTLs, meaning they expire automatically after a defined time. OAuth state remains short-lived and signed. Token blacklists or revocation sets should use TTL equal to token expiry.

Traffic spikes are handled through three layers: API autoscaling, queue-based background processing for expensive persona generation, and backpressure/rate limiting on high-cost endpoints.

API autoscaling means adding more API instances when request volume increases. Queue-based background processing means that expensive work, such as persona generation, is placed into a queue and handled by workers instead of forcing the user-facing API request to wait until all the work is finished. This matters because persona generation may call an LLM provider, write database records, match skills, build files, and publish events. Those operations are slower and more failure-prone than a normal API read request.

Backpressure means the system deliberately slows down or rejects new work when it is already overloaded. Rate limiting means setting a maximum number of requests a user or client can make in a period of time. For high-cost endpoints, such as generation, publishing, OAuth callbacks, or payment actions, V2 should return clear 429 Too Many Requests or 202 Accepted responses instead of allowing unlimited requests to overload Redis, PostgreSQL, the worker pool, or the external LLM provider.

The system should reject excess work predictably instead of letting the database, LLM provider, or worker pool collapse.

Observability

Structured logging uses JSON with required fields:

timestamp
service
environment
level
event
request_id
trace_id
route
status_code
duration_ms
error_class

Structured logging means every important log line follows the same shape. Instead of writing vague messages like something failed, the service emits consistent fields that can be searched and grouped. For example, if an OAuth callback fails, the log should include the service name, environment, route, status code, error class, request ID, and trace ID. That makes it possible to connect a user-visible problem to the exact backend event without exposing secrets.

Core metrics:

request rate by route and status;
p50/p95/p99 latency by route;
5xx rate;
auth failure rate;
OAuth provider failure rate;
persona generation queue depth;
persona generation success/failure count;
Celery task duration and retry count;
Redis connection errors;
database connection pool saturation;
SLO burn rate;
DORA deployment frequency, lead time, CFR, and MTTR.

These metrics are the operational health signals for the platform. Request rate shows demand. Latency shows whether users are waiting too long. Error rate shows whether the service is failing. Queue depth shows whether background workers are falling behind. Database pool saturation shows whether the API is running out of database connections. SLO burn rate shows whether the service is consuming its allowed failure budget too quickly. DORA metrics show whether the team is deploying safely and recovering quickly.

Alerting thresholds:

availability fast burn: 14.4x over 1 hour;
availability slow burn: 5x over 6 hours;
5xx rate above 1% for 5 minutes;
p95 API latency above 500ms for 10 minutes;
CPU above 80% warning and 90% critical;
memory above 80% warning and 90% critical;
disk above 75% warning and 90% critical;
queue depth above expected worker capacity for 10 minutes;
deployment CFR above 15%.

The alert thresholds are intentionally tied to user impact and operational risk. A single slow request should not wake someone up, but sustained high latency or a fast SLO burn should. CPU and memory alerts are useful, but they are secondary signals; the more important question is whether users are experiencing failures or slow responses. Queue depth matters because it warns that background work is piling up before users start complaining that generation is stuck.

Distributed error tracking connects metrics, logs, and traces through trace_id. Grafana derived fields allow jumping from Loki logs into Tempo traces. Application errors include stable error classes but not raw secrets or full user data.

In practical terms, this means an engineer can start from a Grafana latency spike, open the related Loki logs for the same time window, then jump into the Tempo trace for the exact request. The trace shows where time was spent, while the logs explain what happened. This is the difference between knowing "the API is slow" and knowing "persona generation is slow because the LLM call timed out after 30 seconds."

4. Tech Stack Decisions

FastAPI remains appropriate because Anvila is an API-heavy Python backend with strong async support, typed request validation, and easy OpenTelemetry instrumentation. Async support matters because API servers often wait on databases, external APIs, and network calls; async handling helps the server use resources efficiently during that waiting time.

Nginx remains useful at the application edge because it is mature, battle-tested, and efficient at reverse proxying traffic to backend processes. In V2, it should either sit behind an AWS Application Load Balancer or be replaced by a managed ingress layer if the platform moves to containers.

PM2 worked for V1 because it kept the FastAPI staging and production processes running and allowed fast restarts during the task. For V2, PM2 is acceptable for a small VM-based deployment, but a production platform should consider systemd units, containers, or orchestration so process configuration is versioned and less dependent on manual runtime commands.

PostgreSQL remains the primary relational database because Anvila needs transactional integrity for users, personas, OAuth links, refresh tokens, payments, and publishing state. Unique indexes on email and provider subjects protect identity flows from races.

Redis is used for ephemeral coordination because persona generation, rate limiting, short-lived state, Celery queues, and event streams need low-latency TTL-backed storage. "Ephemeral" means temporary. Redis is not the source of truth; if data must survive permanently, it belongs in PostgreSQL or object storage.

Celery remains appropriate for persona generation because LLM calls and GitHub publishing can be slow, retryable, and unsuitable for synchronous request handling. Synchronous request handling would force the user to wait while the server does all the work before returning a response.

Prometheus remains the metrics interface because it is mature, pull-based, and has strong PromQL support for SLO burn-rate alerting. For V2 scale, Prometheus should remote-write to durable long-term storage.

Loki is retained for logs because it integrates tightly with Grafana and is cost-efficient when logs are indexed by labels rather than full text.

Tempo is retained for traces because it is designed for high-volume trace storage with object storage and works well with OpenTelemetry.

Grafana remains the visualization layer because it can unify Prometheus, Loki, and Tempo and supports provisioned dashboards.

OpenTelemetry Collector becomes more central in V2 because it gives a vendor-neutral telemetry pipeline with batching, redaction, retries, memory limits, and routing. Vendor-neutral means the application does not have to be rewritten if the team later changes storage or visualization tools.

Alertmanager remains useful for alert grouping, routing, inhibition, and resolved notifications. V2 strengthens it with ownership metadata and escalation policies.

Node Exporter remains useful for host-level metrics such as CPU, memory, disk, filesystem, and network usage. Blackbox Exporter remains useful for probing public endpoints from outside the application process, because an API can look healthy internally while the public route is broken.

The custom DORA exporter remains useful as a bridge between GitHub Actions and Prometheus. Its V2 responsibility should be narrowed and made more accurate: export deployment timestamps, workflow duration, deployment result, rollback markers, and incident restoration timestamps instead of relying on approximations.

GitHub Actions remains the CI/CD system because it is already the source of deployment workflow events for the Anvila backend. Keeping it reduces migration risk and allows deployment metadata to feed directly into the DORA exporter and release health gate.

Slack remains the primary alert destination because it is where the team already collaborates during incidents. Alertmanager should send structured Slack messages with severity, affected service, current value, dashboard link, runbook link, and resolved/firing status. Slack is not the system of record for incidents, but it is effective for fast human response.

Brevo or another transactional email provider remains useful for lower-urgency notifications and account-related emails. Email should not replace Slack for urgent incidents, but it is useful for user-facing flows, escalation summaries, and non-real-time operational notices.

Gemini remains the LLM provider for persona generation because the existing application already uses it. In V2, LLM calls should stay behind Celery workers and rate limits because they are slower, more expensive, and more failure-prone than normal API reads.

Stripe remains the payment provider where payment features are enabled because it is a mature managed payment platform. The system should not store raw card data. Stripe webhooks should be validated, logged with safe event IDs, and monitored as high-impact integration points.

Terraform remains the infrastructure provisioning tool because the system needs reproducible cloud infrastructure. However, V2 should reduce shell provisioner dependence and move toward immutable images, cloud-init, Ansible, or container orchestration.

AWS remains a reasonable cloud provider because the existing deployment is already on EC2, and AWS provides the missing production pieces: Secrets Manager, IAM, ALB, S3, RDS, ElastiCache, autoscaling, and private networking. ALB means Application Load Balancer, which distributes traffic across healthy backends. S3 provides durable object storage for logs and traces. RDS provides managed PostgreSQL. ElastiCache provides managed Redis.

Grafana Mimir is optional but valuable if metrics retention and scale outgrow a single Prometheus server. The trade-off is operational complexity; the benefit is long-term, horizontally scalable Prometheus-compatible metrics storage.

Proposed V2 Architecture Diagram

The diagram separates the system into six readable paths:

user traffic enters through the public Application Load Balancer and reaches Anvila API replicas;
slow persona generation and GitHub publishing run in Celery workers instead of blocking user requests;
application services send metrics, logs, and traces to an internal OTLP endpoint;
OpenTelemetry Collectors process, redact, batch, and route telemetry to durable storage;
engineers access dashboards through SSO/MFA/RBAC-protected Grafana rather than direct public APIs;
GitHub Actions, the DORA exporter, Alertmanager, and the release health policy close the loop between deployment and reliability.

Conclusion

V1 was successful as a learning-stage observability platform because it proved the full reliability loop: telemetry collection, dashboards, SLOs, alerts, runbooks, incident review, and Game Day validation. It used Anvila as the first real service, which made the work concrete and testable. It was not production-grade because it depended on a single monitoring host, weak identity boundaries, mutable server state, partial DORA accuracy, incomplete data hygiene, and service-specific configuration.

V2 keeps the strongest V1 decisions: LGTM, OpenTelemetry, provisioned dashboards, SLO burn-rate alerting, and DORA visibility. It replaces the fragile parts with highly available collectors, durable telemetry storage, identity-aware access, secrets management, release health gates, and structured ownership metadata. The result is an observability platform that can support Anvila as a real user-facing product and also onboard other applications through the same platform model.

Building a Production-Grade Observability Platform for the Anvila API with LGTM, SLOs, DORA Metrics, and Game Day Testing

Adeolu — Tue, 19 May 2026 00:08:05 +0000

Introduction

For the HNG DevOps Stage 6 task, our team built a production-grade observability and reliability platform for the Anvila API.

The goal was not just to check whether a server was up or down. We needed to build a monitoring system that could help a team understand the health of an application from different angles:

Is the application available?
Is it fast enough for users?
Are requests failing?
Is the server under pressure?
Are deployments healthy?
Can engineers investigate logs, metrics, and traces from one place?
Can alerts reach the team with enough context to act quickly?

To achieve this, we used the LGTM observability stack:

Loki for logs
Grafana for dashboards
Tempo for traces
Prometheus for metrics

We also added Alertmanager for alert routing, Node Exporter for server metrics, Blackbox Exporter for uptime probing, OpenTelemetry Collector for logs and traces, and a custom GitHub Actions DORA exporter.

Prometheus targets all up

Architecture Overview

Our architecture uses a separate monitoring server. The Anvila API continues to run on its application server, while the monitoring stack runs on a dedicated AWS EC2 instance.

The monitoring server collects:

system metrics from Node Exporter
uptime and response time metrics from Blackbox Exporter
application metrics from the FastAPI /metrics endpoint
logs from the app server through OpenTelemetry Collector and Loki
traces from the instrumented FastAPI staging service through OpenTelemetry and Tempo
CI/CD metrics from GitHub Actions through a custom DORA exporter

Grafana connects all these data sources together so that we can view metrics, logs, and traces in one place.

LGTM dashboards list

At a high level, the data flow is:

Anvila API -> OpenTelemetry Collector -> Loki / Tempo
Anvila API /metrics -> Prometheus
Node Exporter -> Prometheus
Blackbox Exporter -> Prometheus
GitHub Actions exporter -> Prometheus
Prometheus / Loki / Tempo -> Grafana
Prometheus alerts -> Alertmanager -> Slack

Why We Chose LGTM Over Managed Alternatives

Managed observability platforms are useful because they reduce operational work. However, for this task, we chose LGTM because we needed to understand and control the full observability pipeline.

LGTM gave us:

full control over metrics, logs, and traces
version-controlled configuration
no dependency on a paid managed observability provider
a better learning experience for how observability systems actually work
the ability to provision dashboards, alert rules, and data sources as code

Prometheus is strong for metrics and alerting. Loki is useful for storing logs in a way that works well with Grafana. Tempo stores distributed traces and helps us understand request flow. Grafana brings all three together in one interface.

This made LGTM a good fit for a platform engineering task where the goal was to build the stack from the ground up.

Infrastructure as Code and Systemd Deployment

One important requirement was that the stack should be reproducible. We used Terraform to provision the monitoring EC2 server and upload the configuration files.

The services are installed and managed with systemd instead of Docker. This was important because the task update said we should not install Docker on the server.

The stack includes systemd services for:

Prometheus
Grafana
Loki
Tempo
Alertmanager
Node Exporter
Blackbox Exporter
OpenTelemetry Collector
DORA exporter

The repository contains:

Terraform files
Prometheus configuration
Alertmanager configuration
Grafana dashboard JSON files
Loki and Tempo configuration
OpenTelemetry Collector configuration
alert rules
runbooks
SLI/SLO documentation
Game Day documentation

Alert rules YAML

Metrics Collection

Metrics are numerical measurements that tell us what is happening in the system.

We collect several types of metrics:

Server Metrics

Node Exporter gives us system-level metrics such as:

CPU usage
memory usage
disk usage
disk I/O
network I/O
system load

These metrics help us understand saturation, which means how full or stressed the system is.

Node Exporter dashboard

Uptime and HTTP Probe Metrics

Blackbox Exporter probes both the staging and production API URLs.

It checks:

whether the endpoint is reachable
whether it returns a successful HTTP response
how long the response takes
SSL certificate expiry

This is important because it measures the service from the outside, closer to how a user or client would experience it.

Blackbox dashboard

Application Metrics

We also added app-level Prometheus metrics to the FastAPI staging application through a /metrics endpoint.

This gives Prometheus real application request metrics such as:

http_requests_total
http_request_duration_seconds
http_requests_in_progress

These metrics help us track request volume, request latency, and request status codes from inside the application itself.

App metrics target up

App request metrics

App latency histogram

Logs and Traces

Metrics tell us that something is wrong. Logs and traces help us understand why.

We used OpenTelemetry Collector to collect application logs and send them to Loki. In Grafana, we can search these logs using Loki queries.

Loki logs

We also instrumented the staging FastAPI service with OpenTelemetry so that it sends traces to Tempo.

A trace shows the path of a request through the application. This is very useful when investigating slow requests, because it helps identify which endpoint or operation caused the delay.

Tempo traces

The Unified Observability dashboard combines metrics, logs, and traces. The idea is that if latency or error rate increases, an engineer can look at the metric, check related logs in Loki, and then inspect traces in Tempo.

Unified observability dashboard

The Four Golden Signals

The Four Golden Signals are a simple way to define service reliability from a user-focused perspective.

They are:

Latency
Traffic
Errors
Saturation

Latency

Latency means how long it takes the service to respond.

For Anvila, we track latency using both Blackbox response time and application request duration metrics.

Example PromQL:

histogram_quantile(
  0.95,
  sum by (le, path) (
    rate(http_request_duration_seconds_bucket{job="anvila-api-staging",status!~"5.."}[5m])
  )
)

This tells us the 95th percentile request latency for successful requests.

Traffic

Traffic means how much demand the service is receiving.

Example PromQL:

sum(rate(http_requests_total{job="anvila-api-staging"}[5m]))

This tells us the number of requests per second.

Errors

Errors measure how many requests are failing.

Example PromQL:

sum(rate(http_requests_total{job="anvila-api-staging",status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_requests_total{job="anvila-api-staging"}[5m])), 1)

This gives us the ratio of failed requests.

Saturation

Saturation shows how full the system is.

For example, CPU usage:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

This goes beyond simply checking if the server is alive. A server can be up but overloaded. The Four Golden Signals help us understand the actual quality of the service.

SLOs and Error Budgets

An SLI is a Service Level Indicator. It is a measurement.

An SLO is a Service Level Objective. It is the target we want to meet.

An error budget is the amount of unreliability we are allowed within a time window.

For Anvila, our main availability SLO is:

99.5% successful probes over 30 days

The error budget calculation is:

(1 - 0.995) * 30 days = 0.15 days = 3.6 hours

This means the service can be unavailable for about 3 hours and 36 minutes in a 30-day window before the error budget is fully consumed.

The SLO dashboard shows:

30-day availability SLI
SLO target
error budget remaining
error budget time remaining
burn rate
7-day and 30-day compliance history

SLO and Error Budget dashboard

Burn Rate Alerting

Burn rate tells us how quickly the service is consuming its error budget.

This is better than simple alerting because it reduces alert fatigue.

For example, a short small failure may not need to wake everyone up. But if the service is burning through its error budget very quickly, the team should respond immediately.

We configured:

Fast Burn alert: critical
Slow Burn alert: warning

The Fast Burn alert means the system is consuming the error budget too quickly and needs immediate action.

The Slow Burn alert means the system may become a serious incident if the trend continues.

DORA Metrics

DORA metrics help connect engineering work to business outcomes.

The four DORA metrics are:

Deployment Frequency
Lead Time for Changes
Change Failure Rate
Mean Time to Restore

These metrics help answer questions such as:

How often do we deploy?
How long does it take for code to reach production?
How often do deployments fail?
How quickly can we recover from incidents?

We built a GitHub Actions DORA exporter that exposes deployment workflow data to Prometheus.

The dashboard shows:

deployment frequency
lead time
change failure rate
MTTR
deployment frequency classification

DORA dashboard

Our current DORA implementation measures commit-to-workflow-completion lead time. A future improvement would be to break it into more detailed sub-intervals such as commit time, workflow trigger time, workflow completion time, and deployment confirmation time.

Alerting and Slack Notifications

Alert rules are stored in version-controlled YAML files, not manually created in the Grafana UI.

We configured alerts for:

host downtime
high CPU
high memory
disk usage
SLO burn rate
change failure rate
MTTR threshold

Alertmanager handles routing and sends alerts to Slack.

The Alertmanager configuration includes:

route grouping by alert name, service, and severity
warning and critical routes
inhibition rules
Slack receiver configuration

Alertmanager config

We also used a structured Slack template. The alert payload includes:

alert name
severity
affected host or target
current value
dashboard link
runbook link
firing or resolved status

Slack template

Here is an example of a firing and resolved Slack notification:

Slack firing and resolved alert

Runbooks and Incident Management

Every alert needs a runbook. A runbook explains what the alert means and what to do first.

Our runbooks include:

what the alert means
likely causes
first investigation steps
how to resolve it
when to roll back
when to escalate

Example runbook:

[SLO burn rate runbook]

We also documented a blameless post-incident review for the simulated latency incident.

The goal of a blameless PIR is not to blame a person. The goal is to understand what happened, what the impact was, what went well, what did not go well, and what should be improved.

Game Day Testing

Game Day is where we intentionally create controlled failures to test whether our monitoring and response system works.

We ran three scenarios.

Scenario 1: Deployment Failure

We created a temporary PR with an intentionally failing GitHub Actions workflow.

The goal was to prove that a deployment or pipeline failure can be detected without touching production.

The PR was closed without merging after screenshots were captured.

Game Day deployment failure PR

Game Day deployment failure check

Scenario 2: Latency Injection

During the latency Game Day, we temporarily added a 2-second delay to the staging root endpoint and generated test requests. The evidence below shows the resulting Tempo traces and the recovery after the delay was removed.

Game Day latency traces

Game Day latency recovery

Scenario 3: Resource Pressure

We created CPU pressure on the monitoring server using controlled background processes.

This triggered a CPU warning alert in Slack. After stopping the pressure, we confirmed the resolved alert was also sent.

Game Day resource pressure trigger

Game Day resource pressure Slack alert

Game Day resource pressure recovery

Toil Identified and Automation

Toil is repetitive manual work that can be automated.

We identified several examples of toil:

1. Manual Dashboard Creation

Creating dashboards manually in Grafana is slow and hard to reproduce.

Automation: All dashboards are stored as JSON and provisioned automatically.

2. Manual Monitoring Server Setup

Installing each service manually would be time-consuming and error prone.

Automation: Terraform creates the monitoring server, and systemd installation scripts configure the observability stack.

3. Vague Alert Investigation

If an alert only says, "CPU high", the engineer still has to search for the server, dashboard, and runbook.

Automation: Alertmanager Slack messages include service, host, metric value, dashboard link, and runbook link.

4. Manual CI/CD Metrics Collection

Checking GitHub Actions manually does not scale.

Automation: The DORA exporter pulls GitHub Actions data and exposes it as Prometheus metrics.

What We Learned

This project helped us understand that observability is more than installing tools.

The tools are only useful when they are connected to reliability goals.

Prometheus helped us collect and alert on metrics. Loki helped us inspect logs. Tempo helped us understand request traces. Grafana helped us bring everything together. Alertmanager helped us route actionable alerts to Slack.

We also learned that good reliability engineering requires:

clear SLIs
realistic SLOs
error budgets
runbooks
incident reviews
controlled failure testing

Current Limitations and Next Steps

The platform is functional, but there are still improvements we can make.

Current limitations:

App-level /metrics is currently verified on staging.
Production is monitored through Blackbox probes and shared host-level metrics.
DORA lead time is currently simplified to commit-to-workflow-completion.
A follow-up PR was opened to normalize unmatched route labels in Prometheus metrics and reduce high-cardinality risk.

Next steps:

promote app-level metrics to production after staging validation
merge the metrics label normalization improvement
add more detailed DORA lead-time sub-intervals
add deployment annotations to Grafana dashboards
continue reviewing SLOs as the service matures

Conclusion

For this task, we built a complete observability platform around the Anvila API using the LGTM stack.

We deployed the monitoring stack with Terraform and systemd, collected metrics, logs, and traces, built dashboards, configured Slack alerts, wrote runbooks, and tested the system through Game Day simulations.

The biggest lesson is that observability is not just about knowing when something is broken. It is about giving the team enough context to understand the problem, respond quickly, and improve the system after every incident.

Repository:

https://github.com/Adeolu1024/anvila-observability-platform

Evidence screenshots:

https://drive.google.com/drive/folders/1v_DiT6XQtq0iJtn5JqxfZecAmxhCdd82?usp=sharing```

Building SwiftDeploy: A Declarative Infrastructure CLI with Observability and Policy Enforcement

Adeolu — Wed, 06 May 2026 20:54:03 +0000

What Is This Project?

SwiftDeploy is a command-line tool that automatically sets up and manages web application deployments. Instead of manually configuring Docker containers, Nginx, and monitoring, you write one file (manifest.yaml) that describes what you want, and the tool builds everything for you.

The project was built in two parts:

Stage 4A: Basic infrastructure generation and container management
Stage 4B: Monitoring, policy enforcement, and audit logging

The Core Idea: Declarative Configuration

In traditional DevOps, you manually write configuration files for each service. With SwiftDeploy, you write a single manifest file, and the tool generates all the configuration files automatically.

manifest.yaml (the only file you edit manually):

services:
  image: swiftdeploy-keeds-api:v1.0.0
  port: 5000
  name: api-service
  mode: stable

nginx:
  image: nginx:alpine
  port: 8080
  proxy_timeout: 30s

network:
  name: swiftdeploy-net
  driver_type: bridge

From this one file, SwiftDeploy generates:

nginx.conf (web server configuration)
docker-compose.yml (container orchestration)
All the settings for monitoring and policy checks

How the Tool Works

The CLI tool (swiftdeploy) has several commands:

Command	What It Does
`init`	Reads manifest.yaml and generates nginx.conf + docker-compose.yml
`validate`	Checks if everything is ready for deployment
`deploy`	Starts all containers and waits for them to be healthy
`promote canary/stable`	Switches between stable and canary modes
`status`	Shows a live dashboard with metrics and policy compliance
`audit`	Generates a report of all events and policy violations
`teardown`	Stops and removes all containers

Stage 4A: The Foundation

API Service

The API service is a Python Flask application that serves:

GET / — Welcome message with current mode and version
GET /healthz — Health check endpoint
POST /chaos — Simulates problems for testing (only in canary mode)

Nginx Proxy

Nginx acts as a reverse proxy, routing all traffic to the API service. It:

Listens on port 8080 (configurable)
Forwards requests to the API service
Returns JSON error responses for 502, 503, 504 errors
Logs all requests in a specific format

Docker Compose

Docker Compose manages all containers:

API service (your application)
Nginx (web server/proxy)
OPA (policy engine, added in Stage 4B)

Stage 4B: Observability and Policy Enforcement

The /metrics Endpoint

The API service now exposes a /metrics endpoint that reports statistics in Prometheus format:

http_requests_total{method="GET",path="/healthz",status_code="200"} 42
http_request_duration_seconds_bucket{le="0.1"} 35
app_uptime_seconds 847
app_mode 0
chaos_active 0

These metrics tell you:

How many requests have been made
How fast responses are
How long the app has been running
Whether you're in stable or canary mode
Whether chaos testing is active

OPA: The Policy Engine

OPA (Open Policy Agent) is a separate container that acts like a security guard. Before you can deploy or promote, the CLI asks OPA: "Is it safe?"

Why use OPA instead of checking directly in the CLI?**

Policies are separate from code — easier to update
If OPA crashes, the CLI still works (just warns you)
OPA is not accessible from the internet (security)

The Two Policies

Infrastructure Policy** (checks before deploy):

Is there enough disk space? (must be > 10GB)
Is the CPU overloaded? (must be < 2.0)

Canary Safety Policy** (checks before promoting to canary):

Is the error rate too high? (must be < 1%)
Is the response time too slow? (P99 must be < 500ms)

Data-Driven Thresholds

The actual numbers (10GB, 2.0, 1%, 500ms) are stored in a separate JSON file, not in the policy code. This means you can change the limits without modifying the policy logic.

thresholds.json:

{
  "infrastructure": {
    "min_disk_gb": 10,
    "max_cpu_load": 2.0
  },
  "canary": {
    "max_error_rate": 0.01,
    "max_p99_latency_ms": 500
  }
}

The Status Dashboard

The swiftdeploy status command shows a live dashboard:

╔═══════════════════════════════════════╗
║     SwiftDeploy Status Dashboard      ║
╠═══════════════════════════════════════╣
║ Mode: canary                         ║
║ Chaos: none                          ║
║ Req/s: 0.98                          ║
║ P99 Latency: 5ms                     ║
║ Error Rate: 0.00%                    ║
║ Uptime: 133s                         ║
╠═══════════════════════════════════════╣
║ Policy Compliance                    ║
║   Infrastructure: PASS               ║
║   Canary Safety:  PASS               ║
╚═══════════════════════════════════════╝

Every time the dashboard refreshes, it saves the data to history.jsonl for the audit trail.

The Audit Report

The swiftdeploy audit command reads history.jsonl and generates audit_report.md with:

A timeline of all events (mode changes, status updates)
A list of policy violations (when checks failed)

Bugs We Fixed

Bug 1: OPA Crashed on Startup

Problem: OPA wouldn't start because of "conflicting rules" error.

Cause: We wrote default deny := [] in the Rego file, which conflicted with deny contains msg if { ... }.

Fix: Removed the default deny := [] line. The contains keyword handles empty sets automatically.

Bug 2: OPA Couldn't Find Threshold Values

Problem: OPA loaded the policy files but couldn't find the threshold numbers.

Cause: The JSON file was in the wrong directory. OPA loads files based on their path structure.

Fix: Moved thresholds.json into a swiftdeploy/ subdirectory so OPA could find it at the correct data path.

Bug 3: Status Dashboard Showed "FAIL" Incorrectly

Problem: The dashboard showed "Infrastructure: FAIL" and "Canary Safety: FAIL" even when everything was within limits.

Cause: The CLI didn't send a timestamp field to OPA. The policy rules need input.timestamp to work. Without it, the rules failed, and the CLI defaulted to "FAIL".

Fix: Added timestamp to all OPA queries.

Bug 4: Nginx Couldn't Find the API Service

Problem: Nginx returned 502 errors saying it couldn't resolve the API service hostname.

Cause: Nginx tried to find the API service at startup, but the container wasn't running yet.

Fix: Added Docker's internal DNS resolver (127.0.0.11) and used a variable for the proxy address. This tells Nginx to look up the hostname when a request comes in, not at startup.

Bug 5: Container Didn't Update After Promoting

Problem: After switching to canary mode, the container was still running in stable mode.

Cause: Using docker compose restart doesn't reload environment variables from the updated docker-compose.yml.

Fix: Changed to docker compose up -d --no-deps <service>, which recreates the container with new settings.

Bug 6: Nginx Permission Denied

Problem: Nginx failed to start with "Permission denied" errors.

Cause: We set user: nginx and removed all Linux capabilities, which prevented Nginx from creating necessary directories.

Fix: Removed the explicit user setting. The official Nginx image handles user switching internally.

Key Design Decisions

Why a Separate OPA Container?

The task required: "The CLI must not make any allow/deny decision itself."

This means:

The CLI asks OPA for permission before every deploy/promote
OPA returns "allowed" or "denied" with a reason
The CLI never makes the decision itself

Benefits:

Policies can be updated without changing the CLI
If OPA is down, the CLI warns but continues
All decisions are logged with reasoning

Why Data-Driven Thresholds?

The task required: "Threshold values must not be hardcoded inside Rego files."

This means the numbers (10GB, 2.0, 1%, 500ms) are in a separate JSON file, not in the policy code. This makes it easy to change limits without touching the policy logic.

Why Separate Policy Files?

The task required: "Organise policies by domain. Each domain owns exactly one question."

This means:

infrastructure.rego only checks disk and CPU
canary.rego only checks error rate and latency
Changing one policy never requires changing another

How the Pieces Fit Together

User runs: swiftdeploy deploy
    │
    ▼
CLI gets host stats (disk, CPU)
    │
    ▼
CLI asks OPA: "Is it safe to deploy?"
    │
    ▼
OPA checks infrastructure policy
    │
    ├── If safe → Start containers
    │
    └── If not safe → Block with reason

User runs: swiftdeploy promote canary
    │
    ▼
CLI scrapes /metrics endpoint
    │
    ▼
CLI calculates error rate and P99 latency
    │
    ▼
CLI asks OPA: "Is it safe to promote?"
    │
    ▼
OPA checks canary safety policy
    │
    ├── If safe → Switch to canary mode
    │
    └── If not safe → Block with reason

Lessons Learned

One file can drive everything**: A single manifest file can generate all the configuration files you need.
Policies should be separate from code**: Using OPA makes policies easier to update and test.
Always handle failures gracefully**: If OPA is down, the CLI warns but continues working.
Simple templates work fine**: You don't need complex template engines for configuration files.
Container recreation vs restart**: Restarting a container doesn't reload environment variables. You need to recreate it.
Docker DNS is important**: Nginx needs to know how to find containers by name, which requires Docker's internal DNS resolver.

Summary

SwiftDeploy is a tool that:

Takes a single manifest file as input
Generates all configuration files automatically
Manages container lifecycle (deploy, promote, teardown)
Enforces safety policies via OPA before deploy/promote
Provides monitoring via /metrics endpoint
Tracks all events in an audit log

The key innovation is that everything is driven by one file, and all safety checks happen automatically before any deployment action.

This article documents the development of SwiftDeploy as part of the HNG Internship DevOps Track, Stage 4.