DEV Community: David de Torres

Prometheus 2.35 – What’s new?

David de Torres — Fri, 06 May 2022 09:04:57 +0000

Prometheus 2.35 was released last month, focusing on a better integration with cloud providers. It also improved the service discovery, performance, and resources usage.

One key change was the migration to Go v1.18. It has brought some changes in the support for TLS 1.0, 1.1, and certificates signed with the SHA1 hash function.

Welcome to this first edition of What's new in Prometheus. We love Prometheus, the de-facto open source standard monitoring tool!

In this article, we will analyze some new features, and the impact they might have on the Prometheus community. Here's our editor's pick:

#10501 Build with Go 1.18

Go 1.18 is now the default version for building Prometheus. This has two major implications in authentication.

TLS 1.0 and 1.1 disabled by default client-side. If you are using one of these, Prometheus now allows defining the minimum TLS version with the min_version config parameter (#10610).
Certificates signed with the SHA-1 hash function are rejected. This doesn't apply to self-signed root certificates.

David Lorite - Integrations Engineer at Sysdig

#10516, #10476, #10365 Better integration with cloud providers

As in previous versions, Prometheus is adding better support to cloud providers. This release includes the libraries to authenticate with Google Cloud Platform (GCP) Kubernetes clusters (e.g. GKE) when configured in the Kubernetes service discovery.

The service discovery is more stable for Azure. Before this release, when any node wasn't reachable, the service discovery for the whole cluster failed. Now, the service discovery continues with the rest of the available nodes. It also includes a metric (prometheus_sd_azure_failures_total) to register when nodes aren't ready during service discovery. You can use this new metric to configure an alert to detect problems in Azure service discovery:

rate(prometheus_sd_azure_failures_total[5m]) > 0

Additionally, the Azure service discovery now includes the ResourceGroup filter. This is helpful to reduce the load of Prometheus service discovery, especially when running several AKS clusters, since ARM API requests are rate-limited in Azure.

Carlos Adiego - Integrations Engineer at Sysdig

#10080, #9570 Enhancements in Kubernetes Service Discovery

This version includes two exciting changes in the Kubernetes service discovery.

The addition of node metadata (name, address, labels annotations) in the targets selected through the pod role (#10080). This allows you to add filters in the relabeling allow list or deny list targets from specific nodes. Even more interesting, add the node labels to all metrics, which later allows you to make groupings by nodes or node labels (or annotations). There is a proposal to also analogously include namespace metadata. To add all the node labels to your metrics, you can use the action labelmap in the relabeling:
```
action: labelmap
regex: __meta_kubernetes_node_(.+)
replacement: 'kubernetes_node_$1'
```
EndpointSlice Kubernetes endpoint promoted to v1 from v1Beta1, as it is deprecated in Kubernetes 1.22.

David de Torres - Engineering manager at Sysdig

#10369 New stats for queries computational cost

Prometheus already had timing statistics for queries. However, as they are based on response time, the same query can have different execution times depending on the load of the system or other parallel queries. That was the motivation to include three new statistics in this version: totalQueryableSamples, totalQueryableSamplesPerStep, and peakSamples.

This is great news for engineers working on optimizing complex queries. Now, they can evaluate and compare the performance of the queries that they build.

Aleksandar Ponjavic - Engineering Manager at Sysdig

#10317, #10500 Optimization of TSDB at start

There are two enhancements in this version that involve the start of the TSDB.

The deletion of temporary files (#10317), something that could be skipped in some scenarios. This way, Prometheus ensures that it's not using more disk space than needed and becomes a better citizen of the cluster.
A more efficient way to read the Write Ahead Log (WAL) at the start-up (#10500). This significantly reduces the time Prometheus needs to update the TSDB after a non-controlled stop.

Jesús Ángel Samitier - Integrations Engineer at Sysdig

#10498 Automatically limit CPU usage to the container CPU limit

Prometheus 2.35 brings a new feature to set the Go's GOMAXPROCS environment variable in Go to the container CPU limits. This feature ensures that the process will never consume more than the limits. This feature is still experimental and can be enabled with --enable-feature=auto-gomaxprocs.

Carlos Arilla, Technical product manager at Sysdig

These were the new features chosen by our team, but there are more. You can find the full list of changes in the official release notes of Prometheus 2.35.

Top key metrics for monitoring MySQL

David de Torres — Fri, 05 Nov 2021 08:39:49 +0000

Monitoring MySQL with Prometheus is easy to do thanks to the MySQL Prometheus Exporter. MySQL doesn’t need an introduction – it’s one of the most used relational databases in the world, and it’s also open-source!

Being such a popular database means that the community behind it is also huge. So don’t worry: you won’t be alone.

In this article, you’ll learn how to start monitoring a MySQL Server with Prometheus – from configuring the exporter to what are the top metrics, with alert examples. If you’re running MySQL with RDS, don’t miss our top 5 key metrics for monitoring Amazon RDS.

Setting things up

Before deploying the exporter in the cluster, you have to create the user and password for the exporter in the database. Enter your MySQL database and execute the following queries:

CREATE USER 'exporter'@'%' IDENTIFIED BY 'YOUR-PASSWORD' WITH MAX_USER_CONNECTIONS 3;

GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';

Substitute the user name and the password in the SQL sentence for your custom ones.

Once you have the user and password for the exporter in the database, you have to create a mysql-exporter.cnf file with the credentials of the exporter. Also, in this file you will include the url to connect your MySQL database in the field host:

[client]
user = exporter
password = "YOUR-PASSWORD"
host=YOUR-DB-IP

In your Kubernetes cluster, create the secret with the mysql-exporter.cnf file. This file will be mounted in the exporter to authenticate with the database:

kubectl create secret generic mysql-exporter \
  --from-file=.my.cnf=./mysql-exporter.cnf

How to deploy the MySQL Prometheus Exporter

To start monitoring MySQL with Prometheus, you’ll need to deploy the MySQL Exporter in your cluster.

To deploy the exporter, you can use the example files provided below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-exporter-exporter
spec:
  selector:
    matchLabels:
      app: mysql-exporter
  replicas: 1
  template:
    metadata:
      labels:
        app: mysql-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9104"
    spec:
      containers:
        - name: mysql-exporter
          image: prom/mysqld-exporter:latest
          args:
            - --config.my-cnf=/tmp/.my.cnf
          ports:
            - containerPort: 9104
          volumeMounts:
            - name: my-cnf
              mountPath: /tmp/.my.cnf
              subPath: .my.cnf
          resources:
            limits:
              memory: "256Mi"
              cpu: "256m"
      volumes:
        - name: my-cnf
          secret:
            defaultMode: 420
            secretName: mysql-exporter
            items:
              - key: .my.cnf
                path: .my.cnf

After applying the Deployment for the exporter, Prometheus will automatically start scraping the MySQL metrics as it already has the standard annotations.

Top 5 metrics to monitor MySQL with Prometheus

#1 Availability

There are two metrics that monitor the availability of your MySQL instance. The first one is mysql_up. If this metric equals zero, the exporter cannot access the database, which can be a symptom of an unhealthy or failed database.

You can create an alert to notify you in case of a database down with the following query:

mysql_up == 0

Also, the metric mysql_global_status_uptime can give you an idea of quick restarts that can pass under the radar of the previous metric. You can create an alert with the following Prometheus query to detect instances that have an uptime of less than half an hour:

mysql_global_status_uptime < 1800

#2 Connections

One of the main sources of errors in databases are connection errors. The metric mysql_global_status_connection_errors_total allows you to detect when the database is generating these errors:

rate(mysql_global_status_connection_errors_total[5m])

One common cause of connection error is the lack of available connections. You can check the percent of available connections used with this Prometheus query:

100 * mysql_global_status_threads_connected / mysql_global_variables_max_connections

If this value is near 100, the database can start refusing new connections. A solution for this can be increasing the number of maximum connections. Before doing that, be sure to check the available number of open files in the system. You can check it with the following Prometheus query:

mysql_global_variables_open_files_limit - mysql_global_variables_innodb_open_files

Want to dig deeper into PromQL? Read our getting started with PromQL guide to learn how Prometheus stores data, and how to use PromQL functions and operators.

#3 Slow queries

Like many databases, MySQL keeps a log for slow queries. The number of entries in this log can be consulted with the metric mysql_global_status_slow_queries. You can create an alert with the following Prometheus query to notify when there are new entries in the slow queries log, which can mean that there is a performance issue:

rate(mysql_global_status_slow_queries[5m])

#4 Cache Hit Rate

MySQL uses in-memory cache to optimize the disk read and write operations. A low hit rate in the cache will impact the performance of the database. This Prometheus query will give you the value of the open tables cache hit rate:

rate(mysql_global_status_table_open_cache_hits[5m]) /
(rate(mysql_global_status_table_open_cache_hits[5m]) + rate(mysql_global_status_table_open_cache_misses[5m]))

Also, you can monitor the buffer pool cache with the following promQL query:

(rate(mysql_global_status_innodb_buffer_pool_read_requests[5m]) -
rate(mysql_global_status_innodb_buffer_pool_reads[5m])) / rate(mysql_global_status_innodb_buffer_pool_read_requests[5m]))

You can tune the buffer size for open tables and pool caches in the MySQL configuration, but keep in mind that doing so will affect the memory usage of your instance. Be sure that there is enough memory. You can check the available memory of the container using cAdvisor metrics:

sum by (namespace,pod,container)((container_memory_usage_bytes{container!="POD",container!=""} -  on (namespace,pod,container) avg by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"})) \* -1 >0 ) / (1024\*1024\*1024)

If your MySQL instance is in AWS RDS, check our blog post on how to monitor your RDS instance

#5 Query types

You can get statistics of the usage of each kind of SELECT query in your MySQL database, using the metric that gives information on each of them:

Full join: mysql_global_status_select_full_join
Full range join: mysql_global_status_select_full_range_join
Select range check (joins without keys that check for key usage after each row): mysql_global_status_select_range_check
Select scan: mysql_global_status_select_scan All of them are counters, so remember to use the function rate in Prometheus queries, like this:

rate(mysql_global_status_select_full_join[5m])

Similarly, you can also have statistics on the use of sorting queries in MySQL:

Sort rows: mysql_global_status_sort_rows
The number of sorts that were done using ranges: mysql_global_status_sort_range
Sort merge (The number of merge passes that the sort algorithm has had to do): mysql_global_status_sort_merge_passes
The number of sorts that were done by scanning the table: mysql_global_status_sort_scan

Next steps

In this article, you learned how easy it is to start monitoring MySQL with Prometheus. You just need the right exporter and configuration.
You also learned the top Prometheus queries you should have in mind when monitoring MySQL with Prometheus.

If you want to get some inspiration for writing Prometheus queries, you can take a look at these examples for monitoring Kubernetes, or download our PromQL getting started cheatsheet.

Trigger a Kubernetes HPA with Prometheus metrics

David de Torres — Thu, 07 Oct 2021 16:18:33 +0000

In this article, you'll learn how to configure Keda to deploy a Kubernetes HPA that uses Prometheus metrics.

The Kubernetes Horizontal Pod Autoscaler can scale pods based on the usage of resources, such as CPU and memory. This is useful in many scenarios, but there are other use cases where more advanced metrics are needed – like the waiting connections in a web server or the latency in an API. Also, in other cases, you might need to combine multiple metrics in a formula or make aggregations.

Keda is an open source project that allows using Prometheus queries, along with multiple other scalers, to scale Kubernetes pods.

Kubernetes HPA

Kubernetes HPA can scale objects by relying on metrics present in one of the Kubernetes metrics API endpoints. You can read more about how Kubernetes HPA works in this article.

Kubernetes HPA is very helpful, but it has two important limitations. The first is that it doesn't allow combining metrics. There are scenarios where combining multiple metrics is convenient, such as calculating the connection usage with the current number of established connections and the maximum number of connections.

The second limitation is the reduced number of metrics that Kubernetes exposes by default: just CPU and memory usage. Sometimes, applications expose more advanced metrics, either by themselves or through exporters. To expose more metrics, you need to publish them in the Kubernetes API metrics endpoint.

Connecting HPA and Prometheus metrics with KEDA

Keda is an open source project that simplifies using Prometheus metrics for Kubernetes HPA.

Installing Keda

The easiest way to install Keda is using Helm.

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

You can check out Keda's documentation page for other installation methods.

How Keda does it?

Keda has a Kubernetes operator that creates both the metrics server and the HPA by defining a Custom Resource Definition (CRD) object called ScaledObject. This object allows you to define what you want to scale and how you want to scale it.

What to scale

Easy: almost anything.

With Keda, you can scale the usual Kubernetes workloads, like Deployments or StatefulSets. Also, you can scale other CRDs – it even has another CRD to scale jobs.

How to scale

This is where the magic is done. You can define triggers in Keda, and there are a lot of different types of them. This article is focused on the Prometheus trigger.

When you set up a Prometheus trigger for a ScaledObject, you define a Prometheus endpoint and a Prometheus query. Keda uses that information to query your Prometheus server and create a metric in the Kubernetes external metrics API. Once you create the ScaledObject, Keda automatically creates the Kubernetes HPA for that.

That's it. You don't need to worry about publishing metrics in the Kubernetes API metrics endpoint or even creating the Kubernetes HPA object!

An example, please

Imagine that you want an HPA for the nginx-server deployment. You want it to scale from 1 to 5 replicas, based on the nginx_connections_waiting metric from the Nginx exporter. If there are more than 500 waiting connections, then you want to schedule a new pod.

Let's create the query to trigger the HPA:

sum(nginx_connections_waiting{job="nginx"})

Easy, right? This query just returns the sum of the nginx_connections_waiting metric value for the nginx job.

Want to learn more about PromQL, the Prometheus query language? Check out the PromQL getting started guide – it also includes a cheatsheet!

Let's define the ScaledObject for this example:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: nginx-scale
 namespace: keda-hpa
spec:
 scaleTargetRef:
   kind: Deployment
   name: nginx-server
 minReplicaCount: 1
 maxReplicaCount: 5
 cooldownPeriod: 30
 pollingInterval: 1
 triggers:
 - type: prometheus
   metadata:
     serverAddress: https://prometheus_server/prometheus
     metricName: nginx_connections_waiting_keda
     query: |
       sum(nginx_connections_waiting{job="nginx"})
     threshold: "500"

Notice the metricName parameter. This is a custom name you set for receiving the value from the query. Keda gets the result of the query and creates the nginx_connections_waiting_keda metric with it. Then, it uses this metric to trigger the escalation. Also, remember to change the serverAddress. :)

Now, you simply need to apply the ScaledObject definition, and the HPA will start working.

What else does Keda offer?

Along with all the benefits of using the metrics in your Prometheus server and applying Prometheus queries to combine them as you want, Keda has additional special features.

It allows you to scale down an object to zero, while the default Kubernetes HPA only allows a minimum value equal or greater than 1.
It allows defining the number of replicas in case it's unable to get the value from the metric, e.g. in an error connection.
It supports a secure connection with Prometheus endpoints with authentication.

Putting it all together

In this article, you learned how to create a Kubernetes HPA easily, without the need to extend the Kubernetes API metrics endpoint. Just by installing and configuring Keda.

In the examples, you also learned how to use a Prometheus PromQL query to trigger the autoscaler.