DEV Community

parsa shahmohammadi
parsa shahmohammadi

Posted on AI-assisted

Linux Server Monitoring with Prometheus, Grafana and Alertmanager (Docker Compose, Alerts That Matter)

Most servers get monitored after the first outage. By then you're reading logs to find out what happened, when an alert could have told you an hour earlier that the disk was filling up.

This guide sets up a small monitoring stack in one Docker Compose file: node_exporter collects host metrics, Prometheus stores them and evaluates alert rules, Alertmanager sends the alerts (Telegram or email), and Grafana draws the dashboards. Then it adds the five alert rules worth starting with, and leaves out the rest.

You need a Linux server with Docker Engine and the Compose plugin (docker compose version should work).

The layout

monitoring/
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── alerts.yml
└── alertmanager/
    └── alertmanager.yml
Enter fullscreen mode Exit fullscreen mode

Step 1: docker-compose.yml

services:
  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus:/etc/prometheus:ro
      - prometheus-data:/prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=30d
    ports:
      - "127.0.0.1:9090:9090"
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager:/etc/alertmanager:ro
    ports:
      - "127.0.0.1:9093:9093"
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.8.1
    command:
      - --path.rootfs=/host
    pid: host
    volumes:
      - /:/host:ro,rslave
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.1.0
    environment:
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin
    secrets:
      - grafana_admin
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"
    restart: unless-stopped

secrets:
  grafana_admin:
    file: ./grafana_admin.txt

volumes:
  prometheus-data:
  grafana-data:
Enter fullscreen mode Exit fullscreen mode

Two choices here matter more than they look:

  • Every port is bound to 127.0.0.1. Prometheus, Alertmanager and node_exporter have no login of their own. Published on 0.0.0.0, they show your whole infrastructure to anyone who finds the IP. Docker also writes its own iptables rules, so a published port can be reachable even when UFW says it's closed. Reach the UIs through an SSH tunnel (ssh -L 3000:127.0.0.1:3000 user@server) or a reverse proxy with authentication.
  • The image versions are pinned. latest changes under you on the next docker compose pull. Upgrade on purpose, after reading the release notes.

Put the Grafana admin password in grafana_admin.txt (openssl rand -base64 24 > grafana_admin.txt && chmod 600 grafana_admin.txt).

Step 2: prometheus.yml

global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
  - /etc/prometheus/alerts.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          server: "web-01"
Enter fullscreen mode Exit fullscreen mode

To watch more servers, run node_exporter on each one and add them as targets. Only let your Prometheus server reach their port 9100, at the firewall.

Step 3: The five alerts worth starting with

Most teams start with too many alerts, get flooded and start ignoring all of them. These five catch the outages that actually happen on most Linux servers. prometheus/alerts.yml:

groups:
  - name: node
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.instance }} is not responding"

      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
        for: 30m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24h"

      - alert: DiskAlmostFull
        expr: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}) * 100 > 90
        for: 10m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is over 90% full"

      - alert: MemoryLow
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "Memory on {{ $labels.instance }} is over 90% used"

      - alert: CpuSaturated
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 15m
        labels: { severity: warning }
        annotations:
          summary: "CPU on {{ $labels.instance }} above 90% for 15 minutes"
Enter fullscreen mode Exit fullscreen mode

Why these:

  • DiskWillFillIn24h is the most useful alert most people don't have. predict_linear looks at the last 6 hours and warns you while there's still time to act, not when the database has already crashed.
  • for: on every rule keeps a short spike (a backup, a deploy) from paging anyone.
  • MemAvailable, not MemFree. Linux uses free memory as cache, so MemFree is always low and means nothing.

Step 4: alertmanager.yml (Telegram or email)

Alertmanager has a native Telegram receiver:

route:
  receiver: telegram
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: telegram
    telegram_configs:
      - bot_token: "123456:ABC-your-bot-token"
        chat_id: -1001234567890
        parse_mode: HTML
        send_resolved: true
Enter fullscreen mode Exit fullscreen mode

Create a bot with @botfather, add it to a group, and get the group's chat_id from https://api.telegram.org/bot<TOKEN>/getUpdates after someone posts in the group. For email, use email_configs with your SMTP server instead.

repeat_interval: 4h matters: an unfixed problem reminds you every 4 hours instead of every 5 minutes.

Step 5: Start it and test an alert

docker compose up -d
docker compose ps
Enter fullscreen mode Exit fullscreen mode
  • Prometheus targets: open http://127.0.0.1:9090/targets through your tunnel. Both jobs should show UP.
  • Grafana: http://127.0.0.1:3000, log in as admin, add Prometheus as a data source (http://prometheus:9090), then import dashboard 1860 ("Node Exporter Full").
  • Test the alert path before you need it: run docker compose stop node-exporter, wait about 3 minutes, and InstanceDown should arrive in Telegram. Start it again and you should get the "resolved" message.

An alert you have never seen fire is an alert you don't know works.

Common mistakes

Mistake What happens Fix
Ports published on 0.0.0.0 Metrics and admin UIs open to the internet, even with UFW on Bind to 127.0.0.1, use a tunnel or proxy with auth
No for: on rules Alerts on every short spike, then everyone ignores them 2 to 15 minutes depending on the rule
Alerting on MemFree Constant false alarms Use MemAvailable
Never testing the alert path First real outage, no message Stop node-exporter once and watch the alert arrive
Monitoring server is the server you monitor It goes down together with everything else Monitor from a second machine, or add an external uptime check

Where to go next

Once host metrics are covered, the next layer is application metrics (Nginx, databases, queues) and logs in one place. If you'd rather have someone else run this, PikoSystem offers observability setup with Grafana and Prometheus and proactive server monitoring, and a free server monitoring review if you want a second opinion on an existing setup.

References: Prometheus: alerting rules, Alertmanager configuration, node_exporter


Parsa Shahmohammadi works on PikoSystem, an IT services and server management team.

Top comments (0)