DEV Community

Cover image for How to Monitor Network Device Health Using SNMP Exporter and Prometheus
Alex Umansky
Alex Umansky

Posted on

How to Monitor Network Device Health Using SNMP Exporter and Prometheus

#networking #monitoring #tutorial #devops

The Art of Monitoring Your Network

Hello dear tech priest readers. My name is Alex Umansky, aka TheBlueDrara, and I welcome you to a small and simple guide I made about monitoring the different devices on my network.

This guide was inspired by a challenge I had to overcome on one of my projects.

I have been tasked to monitor the health state of different devices like servers and switches on my network using Prometheus and Grafana as my main monitoring tools.

So let’s jump right into the fun.

Overview

To begin with, let's start with a short overview of what we are going to learn here.

We will start by shallow diving into what the SNMP protocol is, and since we are using Prometheus as part of our stack for monitoring, we will focus on learning how to use and deploy the SNMP exporter with which we will be able to pull the needed metrics from our devices.

Prerequisites

For the pre-setup, we will need to install and acquire some tools and resources for our work.

We will need:

  • Docker
  • SNMP_exporter Docker image, you can get it here
  • SNMP_generator, using this GitHub repository and create the image from the Dockerfile here
  • Prometheus
  • Grafana
  • snmp-mibs-downloader
  • snmp
  • git 
sudo apt-get install -y docker-ce snmp-mibs-downloader snmp git
docker pull prom/snmp-generator:v0.29.0
docker pull prom/snmp-exporter:v0.28.0
docker pull prom/prometheus:v3.5.0 
docker pull grafana/grafana:12.0.0
Enter fullscreen mode Exit fullscreen mode

Prometheus - monitoring system that collects metrics Grafana - data visualization tool

SNMP exporter - acts as the SNMP manager, sends requests to devices, and collects data

SNMP generator - helps us create the SNMP exporter config file using a textual format

snmp-mibs-downloader - allows us to install the necessary MIB files

snmp - a utility tool for commands like snmpwalk

Before we begin

Before we jump into the setup, it is important to understand how the flow of our tools works.

I promise that I will cover every detail needed in the future, but for now, let's see the flow.

grafana => prometheus => snmp_exporter => hardware devices
Enter fullscreen mode Exit fullscreen mode

We will use Grafana to visualize the data that Prometheus collected.

Prometheus will scrape data from the SNMP exporter that acts as an SNMP manager sending SNMP requests to its target agents (hardware devices).

Again, don't worry, everything will become clearer in the next parts.

Let's break everything up and begin!

In this guide, I won't dive deep into what the SNMP protocol is. For the sake of understanding, I'll simplify the explanation, but if you want to dive deeper into the protocol, there are amazing documents about it that you can read here.

So what is SNMP?

SNMP is quite an old protocol that, I quote, "created a universal language allowing IT teams to decode the operational state of network hardware regardless of manufacturer."

In simple words, we can use this protocol to pull specific metrics and data from our hardware devices regardless of the different vendors.

It works in a server-client architecture, where there is an SNMP manager (our SNMP exporter) and SNMP agents that come almost always preinstalled on modern hardware and just need to be enabled.

SNMP Agent

You need to check for each vendor how to enable SNMPv3 on the device, and you should create a read-only separate user for SNMPv3 metrics and use their credentials for authentication.

Else we won’t be able to pull metrics from the device.

What is SNMP Exporter?

A tool that translates data from network devices using SNMP into a format that Prometheus can understand.

We will need it if we want to use Prometheus as our monitoring tool.

The beauty of this exporter is that you can run the service in one place and just give it a target list of your hardware devices.

The manager will send requests to the agents to collect the data.

The SNMP exporter has two main files that we need to create:

  • snmp.yml - config file that defines which metrics the SNMP exporter should scrape
  • target_list.yml - a list of targets to scrape (hardware devices, switches, servers, etc.)

How to create snmp.yml file

This file holds the configuration that tells the SNMP exporter what metrics we want to pull from the agents.

The issue is that this file needs to be written with OIDs for best results.

Note, OIDs are a numeric tree structure. Each OID is unique and represents a single metric that can be pulled from a device.

To handle this problem, we will use the official SNMP generator.

It will take a bit of setup, but it should be simple.

How to set up the SNMP generator

Start by pulling the official generator repository and building the Docker image.

git pull https://github.com/prometheus/snmp_exporter.git
cd snmp_exporter/generator
Docker build -t <image_name> -f generator/Dockerfile .
Enter fullscreen mode Exit fullscreen mode

Now we need to provide the container with two things:

  • The generator.yaml file
  • The MIB files

Note, MIBs are text files that help us translate human-readable formats into OIDs.

Creating generator.yaml file

The generator config file is a YAML file that configures what metrics we want to pull that is written in a human-readable text format.

Here we just give it a whole module to "walk" (scrape).

We can also specify which exact metrics we want from the module.

In this part, you start to shine — pull the exact metrics you need for your project.

For a simple example, we will use the IF-MIB module.

IF-MIB: network interfaces. SNMPv2-MIB, SNMPv2-SMI, SNMPv2-TC: standard objects, types, counters HOST-RESOURCES-MIB: CPU load, memory, storage.

cd ./snmp_exporter/generator
vim generator.yml
Enter fullscreen mode Exit fullscreen mode

In this example, I want to pull two metrics from the IF-MIB module:

ifOperStatus
ifAdminStatus
Enter fullscreen mode Exit fullscreen mode

The status of admin ports and operational status of ports of a switch.

  if_mib:
    walk:
      - IF-MIB::ifXTable
    overrides:
      IF-MIB::ifOperStatus:          { ignore: false }
      IF-MIB::ifAdminStatus:         { ignore: false }
Enter fullscreen mode Exit fullscreen mode

Downloading MIBs

Now that we have our generator.yaml file, let's install our MIBs.

For that, we will use the snmp-mibs-downloader package.

Run this command to install the default MIBs (there are vendor-specific MIBs with specific metrics).

download-mibs
Enter fullscreen mode Exit fullscreen mode

It will download the MIBs we need to this path: /usr/share/snmp/mibs/

Now let's generate our snmp.yml file.

We will create a directory that will contain our exact MIBs that we used in the generator file.

cd /snmp_exporter/generator
mkdir -p mibs
cp <MIBS> /mibs
Enter fullscreen mode Exit fullscreen mode

Running the container

We will mount the two files to the container and configure two environment variables,

Which MIBs we used and where they are located in the container.

docker run --rm -v "$PWD:/work" -w /work -v "$PWD/mibs:/mibs:ro" -e MIBDIRS="/mibs" -e MIBS="<MIBS>" snmp-gen generate
Enter fullscreen mode Exit fullscreen mode

We should now get an output of an snmp.yml file.

If not, the container will show an error and what is missing to complete the task.

Choosing SNMP version

We are not done yet! One small last thing — what version of SNMP should we use?

To choose what version we will use, we will need to set up an authentication code for our snmp.yml file.

In this guide, we will use SNMPv3 as it’s more secure.

For that, we need to add a small block of code to our snmp.yml file for authentication to our device.

auths:
  switches: # the name of the auth
    version: 3
    username: '<switch_username>'
    security_level: authPriv
    auth_protocol: SHA
    password: '<switch_password>'
    priv_protocol: AES
    priv_password: <PrivateProtocol_password>
Enter fullscreen mode Exit fullscreen mode

Configuring target_list.yaml file

After we finally have the snmp.yaml file, we need a target list — which devices do we want to scrape?

In the target list, we will give a list of targets, what module we used, and what authentication from the snmp.yml file.

- targets: &switches_targets
    - 192.168.0.10
  labels: { module: 'if_mib', auth: 'switches' }
- targets: *switches_targets
Enter fullscreen mode Exit fullscreen mode

Prometheus task

For Prometheus, we need to create a config file that will configure a scrape task from its exporters.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'snmp'
    scrape_interval: 30s
    scrape_timeout: 25s
    metrics_path: /snmp
    file_sd_configs:
      - files:
          - /prometheus/snmp_targets_list.yaml
        refresh_interval: 1m
Enter fullscreen mode Exit fullscreen mode

A small recap of what we did so far and how everything comes together:

  • We installed the tools we need
  • We pulled all the necessary Docker images
  • We generated an snmp.yml using a generator image; for that, we downloaded the MIBs and configured a generator.yaml for our needs
  • We created a target list for the SNMP exporter
  • We enabled SNMPv3 on our hardware and created a read-only user for SNMPv3 metrics
  • We created a Prometheus scraping task

Now let's deploy all our services and make the config files take effect.

The Setup

Using Docker Compose, let's deploy our services with ease.

services:
  prometheus:
    image: prom/prometheus:v3.5.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus-data:/prometheus
    networks:
      - monitoring
    restart: always

  grafana:
    image: grafana/grafana:12.0.0
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    networks:
      - monitoring
    restart: always

  snmp-exporter:
    image: prom/snmp-exporter:v0.28.0
    container_name: snmp-exporter
    network_mode: "host"
    restart: always
    volumes:
      - ./snmp.yml:/etc/snmp_exporter/snmp.yml:ro
    command: --config.file=/etc/snmp_exporter/snmp.yml

volumes:
  grafana-storage:

networks:
  monitoring:
    external: true
Enter fullscreen mode Exit fullscreen mode

Note, that SNMP does not work well with NAT, so in Docker Compose we set it to use Host network mode.

Now we will be able to reach promethuies UI by searching

http://<IP_of_service>:9090
Enter fullscreen mode Exit fullscreen mode

Here we should see that the task we created able to pull the metrics we configured.

How to test if we enabled the SNMP agent

We can use the MIBs to run commands with the Net-SNMP tool like "snmpwalk" to scrape a device manually.

For that, we need to configure the path to the MIBs.

Edit this config file:

/etc/snmp/snmp.conf (system-wide defaults)
Enter fullscreen mode Exit fullscreen mode

And add these lines.

They point to the MIBs directory and specify which MIBs are present and should be used.

mibs +All
mibdirs /usr/share/snmp/mibs:/usr/share/snmp/mibs/ietf:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/net-snmp
Enter fullscreen mode Exit fullscreen mode

If we did everything correctly, we should be able to visit Prometheus and see the task pulling metrics that we configured from our devices.

An easy way to manually check if you can pull data from a device is by running an snmpwalk command:

snmpwalk -v3 -u <switch_username> -l authPriv -a SHA -A '<switch_password>' -x AES -X '<PrivateProtocol_password>' <IP_of_device> -m <Module_name> #IF-MIB
Enter fullscreen mode Exit fullscreen mode

You should see a long output of different metrics from the device.

Grafana

I will not go into building dashboards as it's an art of its own, and this guide is long enough.

I hope you enjoyed this guide. Feel free to comment and give me some feedback to keep learning and improving.

Sharing my small research and how I overcame challenges is a small gesture to the open-source community on my part, and it brings me a bit of joy.

Top comments (0)