Prometheus Observability Platform: Long-term storage

As Prometheus is not so well designed for persisting data, a long-term storage solution is called for. Multiple different products can handle long-term storage for Prometheus metrics for example VictoriaMetrics, Grafana Mimir, Thanos, and M3.

With some of these options, we get the capability to store the data into object storage which is ideal for modern workloads running in Kubernetes as we don’t want to store any persistent data inside our cluster. Object storage can be for example Azure Blob storage or AWS S3. This option however has downsides on performance (compared to block storage), so if you have high performance requirements, you might have to look into block storage options.

In this document, we will be focusing on VictoriaMetrics. It was chosen because it is open-source, highly performant, and all its crucial components are free. VictoriaMetrics can only handle block storage, but it is also very fast due to its simple architecture designed only for local storage. It can be run in single mode or clustered. The central part of the architecture consists of:

• The vmstorage component, which stores the time series data;
• vmselect, used to fetch and merge data from vmstorage; and
• vminsert, which inserts the data into vmstorage nodes.

In the clustered version, data is distributed evenly across the vmstorage nodes by the vminsert component, and the distributed data is then fetched and merged by the vmselect component. In Kubernetes, each of these components will have its own pod and the vmselect and vminsert components will have a service to load balance the traffic. All the vmstorage endpoints (pods) will be connected to the vminsert and vmselect pods.

VictoriaMetrics also has multiple additional features, such as:

• the vmalert component which can be used for alerts about the data, and
• vmagent which can be used as a data ingestion point, and for filtering and re-labelling metrics.
• the vmauth component for simple authentication, which uses credentials from the Authorization header. (You can also use some other component in front of vminsert or vmagent, such as oauth2-proxy to handle authentication.)

We can set up Prometheus to write the data it receives into the long-term storage using the remote_write block in the configuration. If authentication is set up, it also needs to be defined in the remote_write block.

Demo

We will now set up the following architecture with minikube, Prometheus, and VictoriaMetrics. This example assumes that you have completed the steps from Prometheus Observability Platform: Prometheus

First we add the VictoriaMetrics helm chart and install it into the VictoriaMetrics namespace:

helm repo add vm https://victoriametrics.github.io/helm-charts/

helm install vmcluster vm/victoria-metrics-cluster --create-namespace --namespace victoriametrics

We should now see six pods running in the victoriametrics namespace:

kubectl get pods -n victoriametrics
---
NAME                                                           READY   STATUS    RESTARTS   AGE
vmcluster-victoria-metrics-cluster-vminsert-f8d48695c-gqx25    1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vminsert-f8d48695c-t8kcn    1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmselect-77465fb479-42wjs   1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmselect-77465fb479-t2jhp   1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmstorage-0                 1/1     Running   0          58s
vmcluster-victoria-metrics-cluster-vmstorage-1                 1/1     Running   0          58s

To access the vmselect web UI we need to port forward the vmselect service:

kubectl port-forward -n victoriametrics services/vmcluster-victoria-metrics-cluster-vmselect 9090:8481

You can then navigate to http://localhost:9090/select/0/prometheus/vmui to access the vmselect VMUI. The URL is a clustered url with the 0 representing the accountid of the cluster.

To set up remote writing from Prometheus into the VictoriaMetrics we need to update our kube-prometheus-stack deployment with the following:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --namespace prometheus --reuse-values --set 'prometheus.prometheusSpec.remoteWrite[0].url=http://vmcluster-victoria-metrics-cluster-vminsert.victoriametrics.svc.cluster.local:8480/insert/0/prometheus/'

Let’s break down the URL provided above. First, we have the service name for vminsert, which you can find with the following command:

kubectl get svc -n victoriametrics 
---
NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
vmcluster-victoria-metrics-cluster-vminsert    ClusterIP   10.100.63.152   <none>        8480/TCP                     3h32m
vmcluster-victoria-metrics-cluster-vmselect    ClusterIP   10.99.12.151    <none>        8481/TCP                     3h32m
vmcluster-victoria-metrics-cluster-vmstorage   ClusterIP   None            <none>        8482/TCP,8401/TCP,8400/TCP   3h32m

Next we have the namespace victoriametrics. Since Prometheus and VictoriaMetrics are in different namespaces, we have svc.cluster.local followed by the port number of the vminsert service and finally, we have the Prometheus enabled write endpoint with the accountid 0 in it, because we are using the clustered version of VictoriaMetrics.

On the VMUI (http://localhost:9090/select/0/prometheus/vmui) we can now verify that metrics are coming from Prometheus.

kubectl port-forward -n victoriametrics services/vmcluster-victoria-metrics-cluster-vmselect 9090:8481

Navigate to the Query section and insert kube_pod_container_status_restarts_total into the Query field. You should now see approximately the same output as you did from Prometheus.

We have now set up a simple Prometheus and VictoriaMetrics integration

Next part: Prometheus Observability Platform: Alerts