The Docker Networking Gotcha Nobody Warns You About When Building a Self-Hosted Monitoring Stack

#devops #promethus #ansible #grafana

When I automated a full observability stack — Prometheus, Grafana,
Loki, Alertmanager — I hit a networking problem that isn't in
any tutorial.

Here's what happened, the fix, and the full project behind it.

The Stack

Service	Purpose	Port
Node Exporter	Host metrics	9100
Prometheus	Scrape + store	9091
Alertmanager	Alert routing	9093
Loki	Log aggregation	3100
Grafana	Dashboards	3000

Deployed via: Vagrant (VM) → Docker Compose (containers)
→ Ansible (automation)

The Problem

Prometheus runs inside Docker.

Inside Docker, localhost = the container itself.
Not your host machine.

Node Exporter runs on the host at port 9100.
Prometheus inside Docker cannot reach localhost:9100.

Your metrics scrape returns nothing.
No error message. Just empty data.

The Fix

Use the Docker bridge gateway IP instead of localhost:

# prometheus.yml
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['172.17.0.1:9100']  # host via bridge gateway

Then open the port through UFW:

ufw allow from 172.17.0.0/16 to any port 9100

That's it. Prometheus can now reach Node Exporter on the host.

Proactive Alerting (Not Just Dashboards)

- alert: HighCPUUsage
  expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "CPU usage above 85% for 2 minutes"

This moves your stack from passive observation to active
incident response. Alertmanager routes it to Slack, email,
or PagerDuty.

Ansible Makes It Reproducible

- name: Run Prometheus
  community.docker.docker_container:
    name: prometheus
    image: prom/prometheus:latest
    state: started
    recreate: yes
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "./alert_rules.yml:/etc/prometheus/alert_rules.yml"
    ports: ["9091:9090"]