DEV Community

Purushotam Adhikari
Purushotam Adhikari

Posted on

Monitoring Infrastructure with Prometheus and Grafana: A Practical Guide

#infrastructureascode #prometheus #grafana #docker

Monitoring is crucial for maintaining healthy, performant, and reliable infrastructure, especially as systems grow in complexity. Two tools—Prometheus and Grafana—have become industry standards for metrics collection and visualization. In this article, you'll learn how to set up Prometheus and Grafana from scratch, scrape targets, build dashboards, and apply best practices—all with practical steps and code snippets.

What Are Prometheus and Grafana?

  • Prometheus: Open-source systems monitoring and alerting toolkit, designed for reliability and scalability. It collects metrics via HTTP and stores time-series data.
  • Grafana: Visualization and analytics platform that works seamlessly with time-series databases like Prometheus. It lets you build real-time, customizable dashboards.

Use Cases

  • Application and server monitoring
  • Infrastructure and network health tracking
  • Alerting for outages or anomalies
  • Capacity planning

Prerequisites

  • Docker (recommended for local/testing)
  • Basic Linux/Unix CLI knowledge
  • Ports 9090 (Prometheus) and 3000 (Grafana) open

1. Spin Up Prometheus & Grafana with Docker Compose

Create a docker-compose.yml file:

version: '3'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
Enter fullscreen mode Exit fullscreen mode

Add prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
Enter fullscreen mode Exit fullscreen mode

Start both with:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

2. Expose Metrics From a Sample App

Use the popular Node.js express-prometheus-middleware:

npm install express express-prometheus-middleware
Enter fullscreen mode Exit fullscreen mode

Sample server.js:

const express = require('express');
const prometheus = require('express-prometheus-middleware');
const app = express();

app.use(prometheus({ metricsPath: '/metrics' }));

app.get('/', (req, res) => res.send('Hello World!'));

app.listen(3001, () => console.log('Running on 3001'));
Enter fullscreen mode Exit fullscreen mode

Add this target to your prometheus.yml:

  - job_name: 'node-app'
    static_configs:
      - targets: ['host.docker.internal:3001']
Enter fullscreen mode Exit fullscreen mode

Reload Prometheus or restart Docker Compose.

3. Explore Prometheus

Visit http://localhost:9090

  • Try querying: http_requests_total
  • Check Targets: Status → Targets

4. Connect Grafana to Prometheus

  1. Go to http://localhost:3000 (default user: admin / admin)
  2. Add Prometheus as a data source (URL: http://prometheus:9090 if inside Docker network or http://localhost:9090 for local)
  3. Save and test.

5. Build Dashboards in Grafana

  • Click "Create" → "Dashboard"
  • Add a panel.
  • Example PromQL query: http_requests_total
  • Choose "Visualization" type (Graph, Stat, Gauge, etc.)
  • Save dashboard.

6. Set Up Alerts

Prometheus supports rules—edit or add a alert.rules.yml:

groups:
  - name: example
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status="500"}[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
Enter fullscreen mode Exit fullscreen mode

Reference alert rules in your prometheus.yml:

rule_files:
  - "alert.rules.yml"
Enter fullscreen mode Exit fullscreen mode

To receive notifications, configure Alertmanager.

7. Practical Tips & Best Practices

  • Label everything: Job, instance, and custom labels provide deep filtering power.
  • Dashboards: Reuse or import community dashboards from Grafana.com.
  • Persistence: For production, mount persistent storage for Prometheus data.
  • Security: Protect Grafana with strong passwords and enable HTTPS on both services.
  • Automate: Deploy dashboards and configs as code for reproducibility (use Ansible, Terraform, etc.).

Sample Queries

  • Total requests: sum(rate(http_requests_total[1m]))
  • 99th percentile latency: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Wrapping Up

Setting up Prometheus and Grafana is straightforward with Docker and an excellent starting point for robust, production-grade monitoring. From scraping custom metrics to real-time visualization and alerting, this duo forms the backbone of modern observability stacks. Try extending this setup by adding exporters for MySQL, Nginx, or node-level metrics for a full infrastructure view.

Happy monitoring!

Top comments (0)