Project-42

Posted on Mar 11

How to Monitor Root and User-Level Podman Containers with Grafana

#tutorial #grafana #podman #monitoring

I wanted to give it a try to Grafana (again) and in this case, to make sure I can monitor podman containers running from different user namespaces

1. podman-exporter Containers
2. Prometheus Container
3. Grafana Container
4. Deployment
5. Adding a Dashboard to Grafana

1. podman-exporter Containers

To extract the information from the podman containers, will be using 2 different podman-exporter Containers. One for the root podman containers and another one for the user solifugo (uid 1001)

First, create the quadlet files.
podman-exporter needs high privileges to "see" into other containers.
Note the Volume paths specific to Podman.

[|=| raspi in ~ ]$ cat /etc/containers/systemd/podman-exporter.container
[Unit]
Description=Podman Prometheus Exporter
After=network-online.target

[Container]
Image=quay.io/navidys/prometheus-podman-exporter:latest
ContainerName=podman-exporter
Network=host
Environment=CONTAINER_HOST=unix:///run/podman/podman.sock
Volume=/run/podman/podman.sock:/run/podman/podman.sock:ro
User=0
Group=0

[Install]
WantedBy=multi-user.target default.target

[|=| raspi in ~ ]$

To monitor the containers from solifugo user (uid 1001) we setup a second container podman-exporter

[|=| raspi in ~ ]$ cat /etc/containers/systemd/podman-exporter-user1001.container
[Unit]
Description=Podman Prometheus Exporter for User 1001
After=network-online.target

[Container]
Image=quay.io/navidys/prometheus-podman-exporter:latest
ContainerName=podman-exporter-user1001
Network=host
Environment=HOME=/home/solifugo
Environment=CONTAINER_HOST=unix:///run/user/1001/podman/podman.sock
Volume=/run/user/1001/podman/podman.sock:/run/user/1001/podman/podman.sock:ro
Mount=type=tmpfs,tmpfs-size=64M,destination=/home/solifugo
User=1001
Group=1001
Exec=--web.listen-address=:9455

[Install]
WantedBy=multi-user.target default.target

[|=| raspi in ~ ]$

2. Prometheus Container

Now we can create the prometheus deployment.
First, create your prometheus.yml on the host

[|=| raspi in ~ ]$ cat /home/solifugo/pods/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'podman-exporter'
    static_configs:
      - targets: ['localhost:9882']

  - job_name: 'podman-user1001'
    static_configs:
      - targets: ['localhost:9455']

[|=| raspi in ~ ]$

Create the quadlet file making sure we add the yml file as part as a volume

[|=| raspi in ~ ]$ cat /etc/containers/systemd/prometheus.container
[Unit]
Description=Prometheus Metrics Collector
After=network-online.target

[Container]
Image=docker.io/prom/prometheus:latest
ContainerName=prometheus
Network=host
User=1001
PublishPort=9090:9090
Volume=/home/solifugo/pods/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
Volume=/home/solifugo/pods/prometheus/data:/prometheus

[Install]
WantedBy=multi-user.target default.target

[|=| raspi in ~ ]$

3. Grafana Container

We can create now the Grfana deployment
This is the quadlet file

[|=| raspi in ~ ]$ cat /etc/containers/systemd/grafana.container
[Unit]
Description=Grafana Dashboard
# 'After' ensures they start in the right order
After=prometheus.service podman-exporter.service podman-exporter-user1001.service
# 'Requires' ensures if they fail or stop, Grafana knows it's missing its heart
Requires=prometheus.service podman-exporter.service podman-exporter-user1001.service

[Container]
Image=grafana/grafana:latest
ContainerName=grafana
Network=host
PublishPort=3000:3000
#Access to the volume as user solifugo/1001
User=1001
Environment=GF_SECURITY_ADMIN_PASSWORD=yoursecurepassword
Volume=/home/solifugo/pods/grafana:/var/lib/grafana

[Install]
WantedBy=multi-user.target default.target

[|=| raspi in ~ ]$

4. Deployment

Once the files are in /etc/containers/systemd/ we can reload the systemd daemon and start the stack by starting grafana (Because of the dependencies After=, this will pull in Prometheus and the Network automatically):

[|=| raspi in ~ ]$ sudo systemctl daemon-reload
[|=| raspi in ~ ]$ sudo systemctl restart grafana

You should be able to see the targets from the prometheus interface

5. Adding a Dashboard to Grafana

5.1 Login into Grafana

5.2 Add the Prometheus connection

Click the Menu (hamburger icon) > Connections > Data Sources.

Click Add data source and select Prometheus.

In the Connection URL field:

If using Host Networking: Enter http://localhost:9090.

Scroll to the bottom and click Save & test. You should see a green checkmark saying "Data source is working."

5.3 Import the Monitoring Dashboard

Click the Menu > Dashboards.

Click New > Import.

ID: 21559 (Podman Exporter Dashboard) – A dashboard that uses the exact same metrics the exporter is currently providing.

And voila!, we got all of the containers visible:

DEV Community