observability-platform is a full LGTM (Loki, Grafana, Tempo, Prometheus) observability stack built from scratch — containerized with Docker Compose, deployable to Kubernetes, with a custom Python Prometheus exporter, 7 Grafana dashboards provisioned as code, Alertmanager routing, and an OpenTelemetry Collector pipeline.
The application never talks to a monitoring backend directly. It emits telemetry once, to one collector, which owns the routing decisions from there. Everything downstream — storage, correlation, alerting, dashboards — is built and operated as part of this system, not bolted on afterward.
Specifically: the app exports traces to the OTel Collector. Promtail ships its container logs to Loki independently via the Docker socket. Prometheus scrapes metrics directly from exporters (node-exporter, cAdvisor, the custom business exporter). The collector is configured to also handle OTLP metrics and logs, but this stack intentionally mixes push (traces) and scrape (metrics) patterns rather than routing every signal through one pipe.
Below: the architecture, the module boundaries, and the specific tradeoffs behind each one.
Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Observability Stack │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │Prometheus │ │ Loki │ │ Tempo │ │ Alertmanager │ │
│ │(metrics) │ │ (logs) │ │(traces) │ │ (routing) │ │
│ └─────┬────┘ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │
│ │ │ │ │ │
│ ┌─────┴─────────────┴─────────────┴────────────────┴───────┐ │
│ │ Grafana (dashboards) │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
│ │ OTel │ │ Promtail │ │ custom-exporter │ │
│ │ Collector │ │ (log ship) │ │ (Python, /metrics) │ │
│ └──────┬──────┘ └──────┬───────┘ └──────────┬───────────┘ │
│ │ │ │ │
│ ┌──────┴────────────────┴──────────────────────┴───────────┐ │
│ │ Target App (FastAPI) + Load Generator │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Data flows:
- Metrics: Node Exporter (host), cAdvisor (containers), Blackbox Exporter (endpoints), and custom-exporter (business logic) are scraped by Prometheus.
- Logs: Promtail auto-discovers containers via Docker socket and ships logs to Loki.
- Traces: A FastAPI target app instrumented with the OTel SDK pushes spans to the OTel Collector, which routes to Tempo.
- Alerting: Prometheus evaluates alert rules, passes firing events to Alertmanager, which routes to a webhook receiver.
The OpenTelemetry Collector Pipeline
The design decision worth explaining: traces don't go straight from the app to Tempo. They go to the OTel Collector first (port 4317, gRPC), which then forwards to Tempo. Logs and metrics take separate, more direct paths — Promtail ships container logs to Loki, and Prometheus scrapes metrics from exporters. The collector's config also defines metrics and logs pipelines (so it could handle all three signals), but in this stack it's actively used for traces.
That's a deliberate mix, not an oversight: traces benefit from a broker in front of them (batching, resource enrichment, and an easy point to add sampling later), while metrics and logs already have well-established scrape/push patterns in this ecosystem that don't need one.
# otel-collector-config.yaml (simplified)
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
prometheusremotewrite:
endpoint: "http://prometheus:9090/api/v1/write"
otlp/tempo:
endpoint: "tempo:4317"
tls:
insecure: true
loki:
endpoint: "http://loki:3100/loki/api/v1/push"
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
exporters: [loki]
Trace-to-Log Correlation
The real power of the LGTM stack is not having metrics, logs, and traces — it is being able to jump between them.
The target FastAPI app enables OpenTelemetry log correlation, then emits normal application logs:
LoggingInstrumentor().instrument(set_logging_format=True)
logger.info("work request completed")
In Grafana's Loki datasource config, a derivedFields block maps that trace_id to Tempo:
derivedFields:
- datasourceUid: tempo
matcherRegex: "trace_id=(\\w+)"
name: TraceID
url: "$${__value.raw}"
Click any log line in Grafana. The right panel loads the exact trace waterfall. No searching by timestamp, no copying IDs. The correlation is built into the data path.
Custom Python Exporter
Off-the-shelf exporters cover databases and runtimes. But understanding the Prometheus data model means being able to build one from scratch.
The custom exporter is a FastAPI app running prometheus_client on port 9200. A background simulation thread updates metrics at configurable intervals:
# Metric types demonstrated:
active_users_total = Gauge("active_users_total", "Simulated active database users")
jobs_queued_total = Counter("jobs_queued_total", "Total jobs queued")
jobs_processed_total = Counter("jobs_processed_total", "Total jobs processed", ["status"])
webhook_duration = Histogram("webhook_delivery_duration_seconds", "Webhook delivery latency")
webhook_success = Gauge("webhook_delivery_success_ratio", "Webhook success rate")
queue_depth = Gauge("job_queue_depth", "Current job queue depth")
It demonstrates all three core metric types (gauge, counter, histogram), a label cardinality decision on jobs_processed_total (split by status, not by job ID — cardinality stays bounded), and the plain scrape protocol. Prometheus hits /metrics on port 9200 exactly like it would hit any production exporter.
Dashboards as Code
Seven Grafana dashboards are provisioned as JSON files in the dashboards/ directory. On first boot, Grafana loads them automatically — zero manual configuration.
| Dashboard | What It Shows |
|---|---|
| Infrastructure Overview | CPU, memory, disk, network per host |
| Container Metrics | Per-container resource usage from cAdvisor |
| Application SLIs | Request rate, error rate, latency (p50/p95/p99) |
| Custom Business Metrics | From the custom Python exporter |
| Loki Log Explorer | Pre-built LogQL queries |
| Tempo Traces | Trace search + waterfall view |
| Alertmanager | Active alerts, silence status |
The provisioning model: Grafana's datasources/ and dashboards/ config are mounted as volumes locally. The Kubernetes path enables the Grafana Helm chart's dashboard sidecar (sidecar.dashboards.enabled: true in the Helm values), which watches for ConfigMaps labeled grafana_dashboard and loads them automatically — the dashboard ConfigMaps themselves aren't checked into this repo, the sidecar mechanism is. The entire local observability UI is reproducible from a docker compose up.
Alertmanager: Noise Suppression
Alert fatigue is a real operational problem. Alertmanager implements it with inhibition rules:
inhibit_rules:
- source_matchers: [severity="critical"]
target_matchers: [severity="warning"]
equal: ["component", "instance"]
When a host has critical CPU usage, warning-level application latency alerts on the same instance are suppressed. The critical alert is the signal. The warning is noise while the critical is active.
Seven alert rules are defined:
| Rule | Condition | Severity |
|---|---|---|
| HostHighCPU | CPU > 85% for 5m | critical |
| HostDiskFull | Disk > 90% | critical |
| ContainerOOMKilled | OOM kill event | critical |
| AppHighErrorRate | Error rate > 5% for 2m | critical |
| AppHighLatency | p99 > 500ms for 3m | warning |
| BlackboxEndpointDown | Endpoint unreachable for 1m | critical |
| CustomExporterJobQueueHigh | Queue depth > 100 | warning |
Kubernetes Transition
The stack is designed to move from Docker Compose to Kubernetes without architectural changes. In this repo, custom app components are represented with raw manifests, while Prometheus/Grafana/Alertmanager, Loki/Promtail, and Tempo are configured through Helm values:
| Component | K8s / Helm Mapping | Why |
|---|---|---|
| Prometheus | kube-prometheus-stack Helm values with persistent storage | Needs durable time-series data |
| Grafana | kube-prometheus-stack Helm values with dashboard and datasource provisioning | Reproducible UI configuration |
| Alertmanager | kube-prometheus-stack Helm values with webhook routing | Alert grouping and routing |
| Loki | loki-stack Helm values with persistence | Durable log storage |
| Promtail | loki-stack Helm values, chart-managed DaemonSet | One per node, ships container logs |
| Tempo | tempo Helm values with persistence | Trace backend configured for the cluster |
| Postgres | Raw StatefulSet manifest | Local backing store for the target app |
| Target App | Raw Deployment manifest | Instrumented application workload |
| Custom Exporter | Raw Deployment manifest | Stateless business metrics exporter |
| Prometheus RBAC | Raw ServiceAccount, ClusterRole, ClusterRoleBinding | Documents required discovery permissions |
Key lesson: Stateful services on Kubernetes require careful liveness/readiness probe tuning. Under resource pressure, slow initialization triggers cascading probe timeouts and restart loops. Conservative startup delays and resource bounds prevent this.
The full Kubernetes manifests are in kubernetes/manifests/. Helm values for kube-prometheus-stack are in kubernetes/helm-values/.
Top comments (0)