Prometheus Observability Platform: Grafana

Grafana is the industry standard open-source product for visualising metrics stored in a TSDB format, or a variety of other data sources. With Grafana, we can create dashboards, queries, and alerts from the data that we have. With all our metrics in long-term storage, we can use a single data source to access all the metrics from all our infrastructure that uses the metrics platform. This enables easily creating dashboards that aggregate data from multiple different Kubernetes clusters, and enable drilling down to a single resource easily.

Demo

Next, we will set up a Grafana instance into our minikube and use Promxy as the default data source. This example assumes that you have completed the following steps, as the components from those are needed:

• Prometheus Observability Platform: Prometheus
• Prometheus Observability Platform: Long-term storage
• Prometheus Observability Platform: Handling multiple regions

Prerequisites:

• base64

First we start with adding the Grafana Helm chart repository, and installing its contents into the Grafana namespace:

helm repo add grafana https://grafana.github.io/helm-charts

Next, we define Promxy as the data source. In the Helm values file, we need the following block to do this:

datasources.yaml:
  apiVersion: 1
  datasources:
  - name: Promxy
    type: prometheus
    url: "http://promxy.promxy.svc.cluster.local:8082"
    isDefault: true

We are using the svc.cluster.local address for the Promxy service, because all our services are inside the cluster.

I have converted the above into json so that it can be passed to Helm:

helm install grafana grafana/grafana --create-namespace --namespace grafana --set-json 'datasources={"datasources.yaml":{"apiVersion":1,"datasources":[{"name":"Promxy","type":"prometheus","url":"http://promxy.promxy.svc.cluster.local:8082","isDefault":true}]}}'

Next we need to get the admin password for the admin user:

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Now we can port-forward the Grafana service:

kubectl port-forward -n grafana services/grafana 9090:80

Navigate to http://localhost:9090 to access the web UI, and log in with the username admin, and the password acquired in the previous step. From here you can verify that Promxy is set up and acting as the default data source by navigating to Administration -> Data sources -> Promxy, and clicking the Test button at the bottom of the page.

Assuming the test was successful, we can then navigate to the “Explore” item in the menu

and check that we have metrics available in the “metrics explorer” section. Alternatively, we can use the following query to check that metrics are available:

sum(kube_pod_container_status_restarts_total) by (namespace, container)

N.B. you might have to change the time range for the query to get results:

We have now achieved setting up Grafana as the metrics visualisation tool for our metrics platform. This enables us to create dashboards and Grafana alerts for metrics from all sources sending metrics to our long-term storage cluster (or clusters if we have multiple regions) that are queried using Promxy.