Most servers get monitored after the first outage. By then you're reading logs to find out what happened, when an alert could have told you an hour earlier that the disk was filling up.
This guide sets up a small monitoring stack in one Docker Compose file: node_exporter collects host metrics, Prometheus stores them and evaluates alert rules, Alertmanager sends the alerts (Telegram or email), and Grafana draws the dashboards. Then it adds the five alert rules worth starting with, and leaves out the rest.
You need a Linux server with Docker Engine and the Compose plugin (docker compose version should work).
The layout
monitoring/
├── docker-compose.yml
├── prometheus/
│ ├── prometheus.yml
│ └── alerts.yml
└── alertmanager/
└── alertmanager.yml
Step 1: docker-compose.yml
services:
prometheus:
image: prom/prometheus:v2.53.0
volumes:
- ./prometheus:/etc/prometheus:ro
- prometheus-data:/prometheus
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.retention.time=30d
ports:
- "127.0.0.1:9090:9090"
restart: unless-stopped
alertmanager:
image: prom/alertmanager:v0.27.0
volumes:
- ./alertmanager:/etc/alertmanager:ro
ports:
- "127.0.0.1:9093:9093"
restart: unless-stopped
node-exporter:
image: prom/node-exporter:v1.8.1
command:
- --path.rootfs=/host
pid: host
volumes:
- /:/host:ro,rslave
restart: unless-stopped
grafana:
image: grafana/grafana:11.1.0
environment:
- GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin
secrets:
- grafana_admin
volumes:
- grafana-data:/var/lib/grafana
ports:
- "127.0.0.1:3000:3000"
restart: unless-stopped
secrets:
grafana_admin:
file: ./grafana_admin.txt
volumes:
prometheus-data:
grafana-data:
Two choices here matter more than they look:
-
Every port is bound to
127.0.0.1. Prometheus, Alertmanager and node_exporter have no login of their own. Published on0.0.0.0, they show your whole infrastructure to anyone who finds the IP. Docker also writes its own iptables rules, so a published port can be reachable even when UFW says it's closed. Reach the UIs through an SSH tunnel (ssh -L 3000:127.0.0.1:3000 user@server) or a reverse proxy with authentication. -
The image versions are pinned.
latestchanges under you on the nextdocker compose pull. Upgrade on purpose, after reading the release notes.
Put the Grafana admin password in grafana_admin.txt (openssl rand -base64 24 > grafana_admin.txt && chmod 600 grafana_admin.txt).
Step 2: prometheus.yml
global:
scrape_interval: 30s
evaluation_interval: 30s
rule_files:
- /etc/prometheus/alerts.yml
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
- job_name: node
static_configs:
- targets: ["node-exporter:9100"]
labels:
server: "web-01"
To watch more servers, run node_exporter on each one and add them as targets. Only let your Prometheus server reach their port 9100, at the firewall.
Step 3: The five alerts worth starting with
Most teams start with too many alerts, get flooded and start ignoring all of them. These five catch the outages that actually happen on most Linux servers. prometheus/alerts.yml:
groups:
- name: node
rules:
- alert: InstanceDown
expr: up == 0
for: 2m
labels: { severity: critical }
annotations:
summary: "{{ $labels.instance }} is not responding"
- alert: DiskWillFillIn24h
expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
for: 30m
labels: { severity: warning }
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24h"
- alert: DiskAlmostFull
expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) * 100 > 90
for: 10m
labels: { severity: critical }
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is over 90% full"
- alert: MemoryLow
expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
for: 10m
labels: { severity: warning }
annotations:
summary: "Memory on {{ $labels.instance }} is over 90% used"
- alert: CpuSaturated
expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
for: 15m
labels: { severity: warning }
annotations:
summary: "CPU on {{ $labels.instance }} above 90% for 15 minutes"
Why these:
-
DiskWillFillIn24his the most useful alert most people don't have.predict_linearlooks at the last 6 hours and warns you while there's still time to act, not when the database has already crashed. -
for:on every rule keeps a short spike (a backup, a deploy) from paging anyone. -
MemAvailable, notMemFree. Linux uses free memory as cache, soMemFreeis always low and means nothing.
Step 4: alertmanager.yml (Telegram or email)
Alertmanager has a native Telegram receiver:
route:
receiver: telegram
group_by: ["alertname", "instance"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receivers:
- name: telegram
telegram_configs:
- bot_token: "123456:ABC-your-bot-token"
chat_id: -1001234567890
parse_mode: HTML
send_resolved: true
Create a bot with @botfather, add it to a group, and get the group's chat_id from https://api.telegram.org/bot<TOKEN>/getUpdates after someone posts in the group. For email, use email_configs with your SMTP server instead.
repeat_interval: 4h matters: an unfixed problem reminds you every 4 hours instead of every 5 minutes.
Step 5: Start it and test an alert
docker compose up -d
docker compose ps
- Prometheus targets: open
http://127.0.0.1:9090/targetsthrough your tunnel. Both jobs should show UP. - Grafana:
http://127.0.0.1:3000, log in asadmin, add Prometheus as a data source (http://prometheus:9090), then import dashboard 1860 ("Node Exporter Full"). -
Test the alert path before you need it: run
docker compose stop node-exporter, wait about 3 minutes, andInstanceDownshould arrive in Telegram. Start it again and you should get the "resolved" message.
An alert you have never seen fire is an alert you don't know works.
Common mistakes
| Mistake | What happens | Fix |
|---|---|---|
Ports published on 0.0.0.0
|
Metrics and admin UIs open to the internet, even with UFW on | Bind to 127.0.0.1, use a tunnel or proxy with auth |
No for: on rules |
Alerts on every short spike, then everyone ignores them | 2 to 15 minutes depending on the rule |
Alerting on MemFree
|
Constant false alarms | Use MemAvailable
|
| Never testing the alert path | First real outage, no message | Stop node-exporter once and watch the alert arrive |
| Monitoring server is the server you monitor | It goes down together with everything else | Monitor from a second machine, or add an external uptime check |
Where to go next
Once host metrics are covered, the next layer is application metrics (Nginx, databases, queues) and logs in one place. If you'd rather have someone else run this, PikoSystem offers observability setup with Grafana and Prometheus and proactive server monitoring, and a free server monitoring review if you want a second opinion on an existing setup.
References: Prometheus: alerting rules, Alertmanager configuration, node_exporter
Parsa Shahmohammadi works on PikoSystem, an IT services and server management team.
Top comments (0)