DEV Community

Aldo Vázquez
Aldo Vázquez

Posted on

Monitoring Linux instances with Prometheus and Grafana

#devops #performance #go #webdev

If you want to know how is everything going in your servers, you must monitor them, yeah, sometimes read htop or similar commands output can be hard to understand, there are a lot of alternatives, and today I am going to show you how to set up a monitoring suite to always know what is going on your servers, linux instances, and more.

In this tutorial, I will show you how to mount a complete, flexible, and scalable monitoring system, using two great (and free) tools.

What are we going to use?

Prometheus is a powerful, open-source and easy-to-use monitoring solution, it is going to help us to collect and fetch a lot of information from your Linux instances.
Grafana on the other hand, is the most popular technology to create monitoring dashboards in a fast and easy way, also it has a lot of plugins and integrations, therefore, if it exists, you can create a Grafana panel for it.

Let's start!

Installing node_exporter

First of all, we will need to install a Prometheus light-weight daemon in all your servers that you want to monitor, this daemon is node_exporter.
Let's install it.
Connect to your server

$ ssh user@target-instance
Enter fullscreen mode Exit fullscreen mode

Download the node_exporter files

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
Uncompress and rename the folder
$ tar xvf node_exporter-1.0.1.linux-amd64.tar.gz
$ mv node_exporter-1.0.1.linux-amd64 ne_files
Enter fullscreen mode Exit fullscreen mode

Create a new user

$ sudo useradd --no-create-home -s /bin/false node_exporter
Change the node_exporter binary ownership to your new user
$ sudo chown node_exporter:node_exporter ./ne_files/node_exporter
Enter fullscreen mode Exit fullscreen mode

Let's add node_exporter to your local binaries

$ sudo mv ./ne_files/node_exporter /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

Now we are adding node_exporter as a new service
Create a service file for node_exporter with

$ sudo vim /etc/systemd/system/node_exporter.service
Enter fullscreen mode Exit fullscreen mode

Then, add this to the new file, do not forget to save your changes.

[Unit]
Description=node_exporter
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Reload the services daemon to see your new service

$ sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Start it!

$ sudo systemctl start node_exporter
Enter fullscreen mode Exit fullscreen mode

Ok, but how do I know it is working?
You can use:

$ sudo systemctl status node_exporter
Enter fullscreen mode Exit fullscreen mode

Or go to your favorite browser and navigate to
http://<target-instance>:9100/metrics

If everything is working, you should see something similar to

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000108492
go_gc_duration_seconds{quantile="0.25"} 0.000108492
go_gc_duration_seconds{quantile="0.5"} 0.000108492
go_gc_duration_seconds{quantile="0.75"} 0.000108492
go_gc_duration_seconds{quantile="1"} 0.000108492
go_gc_duration_seconds_sum 0.000108492
go_gc_duration_seconds_count 1
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.846384e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.689864e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.445623e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
Enter fullscreen mode Exit fullscreen mode

Remember, you need to repeat this process within every Linux instance that you want to monitor.

Installing Prometheus

Once all your target instances are running node_exporter, we will install Prometheus to scrape and understand your server's metrics. 
You can install it either in your localhost or on another server. In this tutorial, we are installing Prometheus in a new server that will monitor everything. Don't worry if you want to install the monitoring suite in your localhost; both processes are very similar.

Connect to your monitoring server

$ ssh user@<monitoring-server>
Enter fullscreen mode Exit fullscreen mode

Download the Prometheus files from here

$ wget https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz
Enter fullscreen mode Exit fullscreen mode

Now, let's untar and rename the Prometheus files

$ tar xvf prometheus-2.23.0.linux-amd64.tar.gz
$ mv prometheus-2.23.0.linux-amd64 p_files
Enter fullscreen mode Exit fullscreen mode

Create a new user to use Prometheus

$ sudo useradd --no-create-home -s /bin/false prometheus
Enter fullscreen mode Exit fullscreen mode

Change the Prometheus binaries ownership

$ sudo chown prometheus:prometheus p_files/prometheus
$ sudo chown prometheus:prometheus p_files/promtool
Enter fullscreen mode Exit fullscreen mode

Move Prometheus binaries to your local binaries

$ sudo mv p_files/prometheus /usr/local/bin
$ sudo mv p_files/promtool /usr/local/bin
Enter fullscreen mode Exit fullscreen mode

Create folders to store Prometheus configuration

$ sudo mkdir /etc/prometheus
$ sudo mkdir /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode

Do not forget to change the directories ownership

$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
Enter fullscreen mode Exit fullscreen mode

Copy configurations to your new folders

$ sudo cp -r p_files/consoles /etc/prometheus
$ sudo cp -r p_files/console_libraries /etc/prometheus
Enter fullscreen mode Exit fullscreen mode

As you might guess, yes, you have to change the directories ownership

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
Enter fullscreen mode Exit fullscreen mode

Create a prometheus.yml file within /etc/prometheus folder

$ sudo vim /etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Add this to prometheus.yml

global:
 scrape_interval: 10s
scrape_configs:
 - job_name: 'prometheus'
 scrape_interval: 5s
 static_configs:
 - targets: ['<target-instance-1>:9090', '<target-instance-2:9100>']
Enter fullscreen mode Exit fullscreen mode

This is a basic Prometheus configuration file. You must add every instance with the node_exporter daemon running in the "targets" array.
Once the prometheus.yml file is ready, change its ownership

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml
Enter fullscreen mode Exit fullscreen mode

Now, let's create a Prometheus service file

$ sudo vim /etc/systemd/system/prometheus.service
Enter fullscreen mode Exit fullscreen mode

Add the following configuration to prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
 --config.file /etc/prometheus/prometheus.yml \
 --storage.tsdb.path /var/lib/prometheus/ \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Reload the service daemon

$ sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

It's time to run your server!

$ sudo systemctl start prometheus
Enter fullscreen mode Exit fullscreen mode

Check your service status

$ sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode

Now, let's check the Prometheus Web UI

Open your favorite browser and navigate to
http://<monitoring-server-ip>:9090/
Now, you should see something like this,

Prometheus web ui

Let's run a simple query to see if Prometheus is retrieving your instances metrics correctly.
Copy the following query into Prometheus Expression input, then click Execute.

sum by (mode, instance) (irate(node_cpu_seconds_total{job="Prometheus",mode="user"}[30s]))*100/4
Enter fullscreen mode Exit fullscreen mode

Then, you should see a similar output

Query output

As you can see, Prometheus also lists all the instances you added in the targets array before.

Installing Grafana

We have installed node_exporter in your instance to export its metrics.
Then we installed Prometheus on a different server to interpret the instance's metrics. Now, we will install Grafana to create dashboards and charts to make the information easier to read.
The Grafana installation is pretty short. If you want (or need) to install it using the standalone binaries, following the previous steps should give you an idea.
Grafana has a "ready to download and install" deb package; using the following snippet should install Grafana as a new service.

$ sudo apt-get install -y adduser libfontconfig1
$ wget https://dl.grafana.com/oss/release/grafana_7.3.4_amd64.deb
$ sudo dpkg -i grafana_7.3.4_amd64.deb
Enter fullscreen mode Exit fullscreen mode

Once Grafana installation is done, run the service with

$ sudo systemctl start grafana-server
Enter fullscreen mode Exit fullscreen mode

Now open a new tab on your browser and navigate to
http://<monitoring-server>:3000
You should see a login screen.

Grafana login screen

The default username and password are always "admin" after you log in, Grafana will ask you to change your admin password. Remember to use a strong password, please.

Now, let's go to setup a new Data Source to fetch information from our Prometheus service.

Click the gear icon on the left bar, then click the Data Sources section.

Data sources config

Then click the Add data source button

Data Sources page

As you might guess, yes, we will add a new Prometheus data source

Add Prometheus

You should see a similar screen

Data Source configuration

Set the HTTP URL to use your Prometheus service, by default it is set to localhost, which is true in this case, because Grafana and Prometheus are running in the same host.

Then scroll down and click on "Save and test"

Save the data source

Now, navigate to Manage dashboard sections, click the dashboard button on the left bar, then click on "Manage" button.

Prometheus Data Source is already working

In this screen, click on Import

Dashboards Page

You should get this screen, as you can see, there are some ways to import dashboards, click on Upload JSON file.

Import a dashboard JSON file

You can find a lot of examples on internet or, if you prefer, you can create your own custom dashboard. For this tutorial we are using this dashboard, developed by Grafana Labs.
Download the JSON file and select it at the file explorer, then this window should be displayed.
Note: You can find a lot of dashboards here.

Dashboard configuration

Just keep going with the dashboard settings, remember to choose the right Prometheus dashboard at the "Prometheus" dropdown, then, click on import.

Dashboard

And now, your dashboard is ready!
At the upper left corner, is a "Host" dropdown, there you can select any of your instances added to the Prometheus configuration file.
Yes, those are a lot of charts, Grafana allows you to remove or add as many as you want, so, get fun creating your own monitoring dashboard!
Thanks for reading, this is my first tutorial/blog ever! I will appreciate any feedback or comment.

Top comments (3)

Collapse
 
thillsii profile image
thillsii •

Hello Aldo,many thanks for this helpfull, concise and precise post that achieves its purpose, regards

Collapse
 
aldorvv__ profile image
Aldo Vázquez •

Glad you liked it!

Regards

Collapse
 
ajaymanikanta1123 profile image
Guddeti Ajay Manikanta ⭐⭐⭐ •

Welcome to Dev.to community. Here, we have great articles regarding wide range of topics. Please explore and make this community a greater place for all the developers. Thank You.

Read next

testifytech profile image

API Testing – The Most Comprehensive Guide Ever

Steve Wortham -

chintanonweb profile image

Say Goodbye to Unnecessary Re-Renders with React Memo: Step-by-Step Tutorial

chintanonweb -

shani_kumar97 profile image

Linux grep command

Shani Kumar -

hamzakhan profile image

🛠️ React Server Components vs. Client Components: When to Use Which? ⚡

Hamza Khan -