Monitoring Systems with Prometheus and Grafana: A Comprehensive Guide

In this blog we will be walking through a step-by-step guide to build your monitoring system for your devices. In this guide we will be setting up a Prometheus server and Grafana dashboard to analyze your systems resource utilization.
We will be deploying our Prometheus and Grafana in docker containers because they are lightweight and easy to manage.

Why is System Monitoring Important?

System monitoring is crucial for several reasons, as it provides valuable insights and benefits that contribute to the overall health, performance, and security of IT infrastructures and applications. Here are some key reasons why system monitoring is important:

• Early Detection of Issues
• Performance Optimization
• Capacity Planning
• Security and Compliance
• Troubleshooting and Root Cause Analysis
• SLA and KPI Tracking
• Resource Optimization and Cost Reduction
• Trend Analysis and Planning

What is Prometheus

Prometheus is a monitoring and alerting toolkit designed to collect and store time-series data from various sources, such as applications, services, and system components. It is particularly well-suited for monitoring highly dynamic and distributed environments. Prometheus scrapes data from designated endpoints at regular intervals, allowing users to monitor the health and performance of their systems. It uses a query language called PromQL to extract and analyze data. Prometheus also supports alerting based on predefined rules, enabling users to receive notifications when certain conditions are met.

What is Grafana

Grafana is a powerful visualization and analytics platform that works seamlessly with Prometheus and other data sources. It allows users to create custom dashboards and visualizations using various data sources, including Prometheus. Grafana provides a user-friendly interface for exploring and understanding complex data sets. It supports a wide range of charts, graphs, and panels, making it easy to display data in a meaningful way. Grafana is commonly used to create real-time monitoring dashboards, performance reports, and executive summaries, helping teams and organizations gain valuable insights into their systems' health and performance

Setting up the environment for Prometheus and Grafan

• enabling ssh

sudo apt-get install openssh-server
sudo systemctl enable ssh
sudo systemctl start ssh
sudo ufw allow ssh
sudo ufw enable
sudo ufw status

• installing docker

sudo apt install docker.io

Configuring Prometheus and Grafana server on ubuntu

• create a directory to store all the configuration files which will be mounted to the docker container

cd /opt/
mkdir prometheus
cd prometheus
touch prometheus.yml
touch alerts.rules

alerts rules and alert manager will be discussed in the next post

• create docker container for prometheus

docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus

• creating a node exporter a node exporter helps in retrieving the metrics of a Linux based system and send it to prometheus

docker run -d -p 9100:9100 -v --name node-exporter prom/node-exporter

• create docker container for grafana

docker run -d --name=grafana -p 3000:3000 grafana/grafana

now open any browser and search for ubuntu_ip:9090, ubuntu_ip:3000 and ubuntu_ip:9100/metrics for checking whether the prometheus, grafana and node exporter are running or not
ubuntu_ip is the IP address of the machine where the docker container is created

if you want to monitor your windows machine you can install a windows exporter from here. After installing you can start the service to start the windows exporter

Configuring Prometheus Server

vim /opt/prometheus/prometheus.yml

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['ubuntu_id:9100']
  # ubuntu_ip is the IP address where node exporter is installed

  # for windows exporter
  - job_name: 'win-exporter'
    static_configs:
      - targets: ['windows_ip:9182']
  # windows_ip is the IP address of the window machine

docker restart prometheus

• now you will be able to use PromQL to gain insights of you Linux machine by visiting ubuntu_ip:9090

Configuring Grafana Dashboard

open ubuntu_ip:3000 on your browser

now use the default username and password admin and admin and after that add prometheus to your datasource

After adding the datasource build a dashboard for your node exporter using the following library Grafana Dashboards

Thanks for reading the blog I hope it was helpful for you. In the next blog we will be configuring our Prometheus Server to send alerts based on some rules.

Comments

[Kudzai Murimi]

Thanks a lot!