Julien Acroute for Camptocamp Infrastructure Solutions

Posted on Mar 26, 2021 • Edited on May 28, 2021

Intro to Prometheus for Developers

#prometheus #kubernetes #observability #monitoring

There are many tutorials to install Prometheus in a Kubernetes cluster. Some of them explain how to install an exporter for mainstream apps. See for example the following video:

Step by Step tutorial to monitor third-party apps using Prometheus (MongoDB example 🍃)

TechWorld with Nana ・ Sep 25 '20

#prometheus #devops #kubernetes #tutorial

But what about monitoring your own application? You may already have metrics for:

Cluster nodes with Node Exporter
Workload in the cluster: Pod metrics, aggregation by namespace
Third party apps: Load balancer, Database, Proxy

The core component, the application itself, should also be monitored. Some metrics can be retrieved from the middleware by using a dedicated exporter:

HTTP requests: total requests, duration, sessions
Memory usage: Garbage collector, cache

We still only have technical metrics about compute resources and their usage. This kind of metrics can be retrieved from any web project, as there always are HTTP requests, DB connection pool, memory, and CPU usage.

The most important indicators in your application are the ones that differ from other apps. For example, a web store app might have the following elements:

cart: total active cart count, total items in active carts
item: number of items available, number of items with no stock
customer: number of accounts, number of accounts with a connection in the last 3 months

You can build metrics within your app to be able to see the evolution of the cart, item, or customer accounts. Each application is unique, so only the people using or contributing to the application know which metric can be created to improve the level of observability of the application.

This series of blog posts will focus on adding high level metrics to existing applications. We will:

Modify an application to generate metrics in the Prometheus format
Connect Prometheus to the application
Create Grafana dashboard and configure alerts
Package monitoring stuff with the whole application using a Helm chart

External Exporter or Integrated in Application?

It's always better to include code that will generate metrics within the application. But sometimes you either:

cannot modify the application code
or the application already exposes some metrics, though not in the Prometheus format.

In this case, you may have to create a separate application that will compute metrics based on information retrieved from the main application, and act as an adapter.

What needs to be generated?

You need to generate one or more metrics. For each metric, the exporter needs to output:

a description
the name of the metrics
an optional list of labels
a decimal value

Metric Labels

Labels are used to create multiple metrics with the same name which refer to different elements. Labels add dimensions to metrics. With labels, you can have multiple values for the same timestamp.

For example, a metric regarding disk usage can look like this:

# HELP disk_usage_percent Usage of disk in percent (0-100)
# TYPE disk_usage_percent gauge
disk_usage_percent 63.7

If there is more than one partition, you can use labels to identify partitions, but keep the same metrics name:

# HELP disk_usage_percent Usage of disk in percent (0-100)
# TYPE disk_usage_percent gauge
disk_usage_percent{partition="/"} 63.7
disk_usage_percent{partition="/var"} 37.2
disk_usage_percent{partition="/tmp"} 12.5

For HTTP request metrics, you can use labels for:

the HTTP method: method="POST"
the path: path="/assets/script.js"
the browser type: browser="Firefox"

For a store, you can build a sales-related metric and use labels for:

the item category: category="Tools", category="Garden and Outdoor"
the brand: brand="Makita", brand="Weber"

You can also create a metric for the stock and use the same labels. This way, when displaying metrics, you will be able to correlate sales with stock of a specific category using labels.

Labels can be viewed as additional dimensions for a metric.

Every time series is uniquely identified by its metric name and optional
key-value pairs called labels.

Prometheus Text Format

Let's talk about the Prometheus format. It's a text based format:

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="get",code="200"} 1234027
http_requests_total{method="post",code="200"} 1027
http_requests_total{method="post",code="400"} 3

Lines starting with # define metric metadata:

# HELP <metric name> <description>: Human readable description
# TYPE <metric name> [counter|gauge|histogram|summary]: Metric type

Most metrics use counter or gauge:

counter metrics are always incremental, cumulative, e.g.: error count, total requests
gauge metrics can increase and decrease, e.g.: connected customers, disk usage

The curly brackets contain a comma-separated collection of labels:

method="get"
code="200"

Finally, separated by a space, is the value.

So for our webstore, we can generate the following metrics:

# HELP disk_usage_percent Usage of disk in percent (0-100)
# TYPE disk_usage_percent gauge
disk_usage_percent{partition="/"} 63.4
disk_usage_percent{partition="/var"} 37.6
disk_usage_percent{partition="/tmp"} 12.3

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET", code="200"} 1234027
http_requests_total{method="POST", code="200"} 1027
http_requests_total{method="POST", code="400"} 3

# HELP sales_total The total number of sales.
# TYPE sales_total counter
sales_total{category="Tools", brand="Makita"} 163376
sales_total{category="Tools", brand="Bosh"} 298425
sales_total{category="Garden and Outdoor", brand="Weber"} 18346
sales_total{category="Garden and Outdoor", brand="Hyundai"} 163376

# HELP stock The number of stock items.
# TYPE stock gauge
stock{category="Tools", brand="Makita"} 234
stock{category="Tools", brand="Bosh"} 456
stock{category="Garden and Outdoor", brand="Weber"} 27
stock{category="Garden and Outdoor", brand="Hyundai"} 45

Note that in this example, we mix metrics from different layers:

System metrics: disk usage
Middleware metrics: HTTP requests
Application metrics: stocks and sales

For system metrics, it is recommended to use a dedicated exporter like
Node Exporter. For middleware, you should be able to find an exporter for your needs: Django, RabbitMQ, Redis, Java JMX, PostgreSQL.

In the next blog post, we will start adding metrics to an existing application.