DEV Community: Deepanshu Anand

Alerts and Alert Manager in Prometheus

Deepanshu Anand — Tue, 01 Aug 2023 04:42:30 +0000

In the last blog we walked through the process of setting up a Prometheus server with Grafana dashboard. In this blog we will be setting up the alertmanager in the prometheus server.

Structure of an alertmanager and the prometheus server

Defining rules for prometheus server

In the last blog we created a alerts.rules config files, this file will contain all the rules for our prometheus server

cd /opt/prometheus/
vim alerts.rules

groups:
  - name: Linux-disk-space
    rules:
     - alert: Linux-DiskSpaceAlert
       expr:  (node_filesystem_avail_bytes/node_filesystem_size_bytes)*100 < 20
       for: 5m
       annotations:
        summary: "Disk space is running low"
        description: 'The available disk space on {{ $labels.instance }} is less than 80percent'


  - name: Linux-memory-utilisation
    rules:
     - alert: Linux-MemoryUsage
       expr: ((node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100 ) < 20
       for: 2m
       annotations:
        Summary: "High RAM Usage"
        description: 'Memory usage on {{ $labels.instance }} is greater than 80percent'

  - name: Linux-up
    rules:
     - alert: Linux-up-status
       expr: up{job='node-exporter'} < 1
       for: 2m
       annotations:
        Summary: "Linux machine/exporter down"
        description: '{{ $labels.instance }} metrics could not be fetched machine may be down'

  - name: Windows-memory-utilisation
    rules:
     - alert: Windows-MemoryUsage
       expr:  ((windows_os_physical_memory_free_bytes/windows_cs_physical_memory_bytes)*100 ) < 20
       for: 2m
       annotations:
        Summary: "High RAM Usage"
        description: 'RAM usage on {{ $labels.instance }} is higher than 80 percent'

  - name: Windows-diskspace-utilisation
    rules:
     - alert: Windows-diskspace-utilisation
       expr: (windows_logical_disk_free_bytes/windows_logical_disk_size_bytes)*100 < 20
       for: 2m
       annotations:
        Summary: "Disk Space running low"
        description: 'Diskspace utilisation on {{ $labels.instance }} is higher than 80 percent'

  - name: Windows-up
    rules:
     - alert: Windows-up-status
       expr: up{job='win-exporter'} < 1
       for: 2m
       annotations:
        Summary: "Windows machine/exporter down"
        description: '{{ $labels.instance }} metrics could not be fetched machine may be down'

Setting the alertmanager

for this walkthrough we will be setting up gmail alerts using gmail smtp server

1. setting up gmail smtp server

To set up a gmail smtp server for prometheus we need a gmail account say sender@gmail.com on which 2-step verification is enabled

your app password is your secret-key which will be used in alertmanager.yml as auth_password

2. config file for alertmanager

cd /opt/prometheus/
touch alertmanager.yml
vim alertmanager.yml

global:
  resolve_timeout: 1m

route:
  receiver: 'gmail-alert-notification'
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1m

# sender@gmail is the gmail account on which you have generated your app password
receivers:
- name: 'gmail-alert-notification'
  email_configs:
    - to: 'receiver@gmail.com'
      from: 'sender@gmail.com'
      smarthost: smtp.gmail.com:587
      auth_username: 'sender@gmail.com'
      auth_identity: 'sender@gmail.com'
      auth_password: 'secret-key'

3. installing the alertmanager docker image

We will be installing the docker image of alertmanager on the same machine on which we installed the prometheus server

docker run -d -p 9093:9093 -v /opt/prometheus/alertmanager.yml:/etc/alertmanager/alertmanager.yml --restart unless-stopped --name alertmanager prom/alertmanager

verify that alertmanager is running by visiting alertmanager-ip:9093 in your browser

Adding rules and alertmanager in the prometheus

cd /opt/prometheus/
vim prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['ubuntu_id:9100']

  - job_name: 'win-exporter'
    static_configs:
      - targets: ['windows_ip:9182']

rule_files:
  - "/etc/prometheus/alerts.rules"

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - "ubuntu_ip:9093"
# ubuntu_ip is the IP on which the alertmanager container is installed, use localcost:9093 if same as prometheus server

# if prometheus docker is already running run the following command
docker restart prometheus

# if not run the following
docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus

Your prometheus and alertmanager is ready and can be verified by visiting prometheus_ip:9090 and alertmanager_ip:9093 on your browser

I hope this blog helped you understand how to setup prometheus with alertmanager. Stay connected for more such blogs

Monitoring Systems with Prometheus and Grafana: A Comprehensive Guide

Deepanshu Anand — Tue, 25 Jul 2023 12:22:21 +0000

In this blog we will be walking through a step-by-step guide to build your monitoring system for your devices. In this guide we will be setting up a Prometheus server and Grafana dashboard to analyze your systems resource utilization.
We will be deploying our Prometheus and Grafana in docker containers because they are lightweight and easy to manage.

Why is System Monitoring Important?

System monitoring is crucial for several reasons, as it provides valuable insights and benefits that contribute to the overall health, performance, and security of IT infrastructures and applications. Here are some key reasons why system monitoring is important:

Early Detection of Issues
Performance Optimization
Capacity Planning
Security and Compliance
Troubleshooting and Root Cause Analysis
SLA and KPI Tracking
Resource Optimization and Cost Reduction
Trend Analysis and Planning

What is Prometheus

Prometheus is a monitoring and alerting toolkit designed to collect and store time-series data from various sources, such as applications, services, and system components. It is particularly well-suited for monitoring highly dynamic and distributed environments. Prometheus scrapes data from designated endpoints at regular intervals, allowing users to monitor the health and performance of their systems. It uses a query language called PromQL to extract and analyze data. Prometheus also supports alerting based on predefined rules, enabling users to receive notifications when certain conditions are met.

What is Grafana

Grafana is a powerful visualization and analytics platform that works seamlessly with Prometheus and other data sources. It allows users to create custom dashboards and visualizations using various data sources, including Prometheus. Grafana provides a user-friendly interface for exploring and understanding complex data sets. It supports a wide range of charts, graphs, and panels, making it easy to display data in a meaningful way. Grafana is commonly used to create real-time monitoring dashboards, performance reports, and executive summaries, helping teams and organizations gain valuable insights into their systems' health and performance

Setting up the environment for Prometheus and Grafan

enabling ssh

sudo apt-get install openssh-server
sudo systemctl enable ssh
sudo systemctl start ssh
sudo ufw allow ssh
sudo ufw enable
sudo ufw status

installing docker

sudo apt install docker.io

Configuring Prometheus and Grafana server on ubuntu

create a directory to store all the configuration files which will be mounted to the docker container

cd /opt/
mkdir prometheus
cd prometheus
touch prometheus.yml
touch alerts.rules

alerts rules and alert manager will be discussed in the next post

create docker container for prometheus

docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/alerts.rules:/etc/prometheus/alerts.rules --restart unless-stopped --name prometheus prom/prometheus

creating a node exporter a node exporter helps in retrieving the metrics of a Linux based system and send it to prometheus

docker run -d -p 9100:9100 -v --name node-exporter prom/node-exporter

create docker container for grafana

docker run -d --name=grafana -p 3000:3000 grafana/grafana

now open any browser and search for ubuntu_ip:9090, ubuntu_ip:3000 and ubuntu_ip:9100/metrics for checking whether the prometheus, grafana and node exporter are running or not
ubuntu_ip is the IP address of the machine where the docker container is created

if you want to monitor your windows machine you can install a windows exporter from here. After installing you can start the service to start the windows exporter

Configuring Prometheus Server

vim /opt/prometheus/prometheus.yml

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['ubuntu_id:9100']
  # ubuntu_ip is the IP address where node exporter is installed

  # for windows exporter
  - job_name: 'win-exporter'
    static_configs:
      - targets: ['windows_ip:9182']
  # windows_ip is the IP address of the window machine

docker restart prometheus

now you will be able to use PromQL to gain insights of you Linux machine by visiting ubuntu_ip:9090

Configuring Grafana Dashboard

open ubuntu_ip:3000 on your browser

now use the default username and password admin and admin and after that add prometheus to your datasource

After adding the datasource build a dashboard for your node exporter using the following library Grafana Dashboards

Thanks for reading the blog I hope it was helpful for you. In the next blog we will be configuring our Prometheus Server to send alerts based on some rules.