DEV Community

Cover image for Prometheus - HTTPS & Authentication - Part 4
Unni P
Unni P

Posted on • Originally published at iamunnip.hashnode.dev

Prometheus - HTTPS & Authentication - Part 4

#prometheus #observability #cloudnative #opensource

In this article, we will look how we can configure HTTPS and Authentication on both Prometheus and Node Exporter

Prerequisites

In my previous article, we looked at how we can set up Prometheus and Node Exporter as systemd services on an Ubuntu instance.

Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

In this article, we will look how to install and configure Prometheus and Node Exporter on Amazon EC2 Ubuntu instance

favicon iamunnip.hashnode.dev

But in the above setup, we were accessing the Prometheus expression browser and Node Exporter metrics endpoints via HTTP and there was no authentication enabled.

We are going to address these issues in this article.

HTTPS

Node Exporter

Create a new directory for storing the Node Exporter configuration file and change its ownership to "node_exporter" user

$ sudo mkdir -p /etc/node_exporter

$ sudo chown node_exporter:node_exporter /etc/node_exporter
Enter fullscreen mode Exit fullscreen mode

Create a configuration file for Node Exporter and change its ownership to "node_exporter" user

$ sudo touch /etc/node_exporter/node_exporter.yml

$ sudo chown node_exporter:node_exporter /etc/node_exporter/node_exporter.yml
Enter fullscreen mode Exit fullscreen mode

Generate a certificate and key using OpenSSL

$ sudo openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout prom.key -out prom.crt -subj "/C=US/ST=California/L=Oakland/O=MyOrg/CN=localhost" -addext "subjectAltName = DNS:localhost"

$ ls
prom.crt  prom.key
Enter fullscreen mode Exit fullscreen mode

Copy the certificate and key files to the Node Exporter configuration directory and change their ownership to "node_exporter" user

$ sudo cp prom.* /etc/node_exporter

$ sudo chown node_exporter:node_exporter prom.crt

$ sudo chown node_exporter:node_exporter prom.key
Enter fullscreen mode Exit fullscreen mode

Add the tls_server_config details to the configuration file

$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
Enter fullscreen mode Exit fullscreen mode

Update the systemd unit file of node_exporter service to include the above configuration file

$ sudo vi /etc/node_exporter/node_exporter.yml

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
        --web.config.file /etc/node_exporter/node_exporter.yml

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Restart the node_exporter service and verify its status

$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter
Enter fullscreen mode Exit fullscreen mode

The metrics endpoint is now accessible via HTTPS

node-2

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down

prom-7

Prometheus

Copy the certificate and key files to the Prometheus configuration directory and change its ownership to "prometheus" user

$ sudo cp prom.* /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prom.crt

$ sudo chown prometheus:prometheus /etc/prometheus/prom.key
Enter fullscreen mode Exit fullscreen mode

Update the Prometheus configuration file to include scheme and tls_config for the "node_exporter" job

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
    static_configs:
      - targets: ["172.31.81.113:9100"]
Enter fullscreen mode Exit fullscreen mode

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Enter fullscreen mode Exit fullscreen mode

Restart the prometheus service to take effect the new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up

prom-8

Now we have enabled secure communication between the Prometheus server and Node Exporter but still our Prometheus expression browser is using an HTTP connection

Create a new configuration file for configuring HTTPS connection and change its ownership to "prometheus" user

$ sudo touch /etc/prometheus/webconfig.yml

$ sudo chown prometheus:prometheus /etc/prometheus/webconfig.yml
Enter fullscreen mode Exit fullscreen mode

Add the tls_server_config details to the newly created configuration file

$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
Enter fullscreen mode Exit fullscreen mode

Update the systemd unit file of prometheus service to include the above configuration file

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries \
    --web.config.file /etc/prometheus/webconfig.yml

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Now we can access the Prometheus expression browser using HTTPS

prom-11

Authentication

Install the apache2 utils package to generate a password

$ sudo apt update

$ sudo apt install apache2-utils
Enter fullscreen mode Exit fullscreen mode

Generate the password using the htpasswd tool

$ htpasswd -nBC 16 admin | tr -d ':\n'
New password:
Re-type new password:
admin$2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
Enter fullscreen mode Exit fullscreen mode

Node Exporter

Update the Node Exporter configuration file to include basic authentication

$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
Enter fullscreen mode Exit fullscreen mode

Restart the node_exporter service and verify its status

$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter
Enter fullscreen mode Exit fullscreen mode

Now access the Node Exporter metrics endpoint and it will show a login prompt

node-3

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down

prom-9

Prometheus

Update the Prometheus configuration file to include basic authentication for the "node_exporter" job

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
      basic_auth:
        username: admin
        password: Password!
    static_configs:
      - targets: ["172.31.81.113:9100"]
Enter fullscreen mode Exit fullscreen mode

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax
Enter fullscreen mode Exit fullscreen mode

Restart the prometheus service to effect new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode

Open the Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up

prom-10

Now we have enabled basic authentication between the Prometheus server and Node Exporter but we need to enable authentication for the Prometheus server

Update the below configuration file to include basic authentication on the Prometheus server

$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O
Enter fullscreen mode Exit fullscreen mode

Restart the prometheus service for the new changes and check its status

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus
Enter fullscreen mode Exit fullscreen mode

Now access the Prometheus expression browser and it will show a login prompt

prom-12

That's all for now

Reference

https://prometheus.io/docs/prometheus/latest/configuration/https/

https://kodekloud.com/courses/prometheus-certified-associate-pca/

Top comments (0)

Read next

anmolbaranwal profile image

All the tools I need to build a perfect AI app.

Anmol Baranwal -

luizhlelis profile image

OpenTelemetry Journey #00 - Introduction to OpenTelemetry

Luiz Lelis -

nikl profile image

How Redis Addressed Our Dynamic Task Scheduling and Concurrent Execution Issues w Celery & Postgres

Nik L. -

samchon profile image

[Typia] executable demo site of 20,000x faster validator (serializer)

Jeongho Nam -