DEV Community: Ryan

Getting started with grafana and prometheus for kubernetes metrics

Ryan — Thu, 03 Sep 2020 07:56:59 +0000

In the previous post, we've gotten Grafana up and running with a cloudwatch datasource. While it provides us with many insights on AWS resources, it doesn't tell us how our applications are doing in our Kubernetes cluster. Knowing the resources our applications consume can help prevent disasters, such as when applications consume all the RAM on the node, causing it to no longer function, and we now have dead nodes and applications.

For us to view the metrics on our Grafana dashboard, we can integrate it into a Prometheus datasource, and have Prometheus collect metrics from our nodes and applications. We will deploy Prometheus using helm, and explain more along the way.

Requirements
The quick 5-minute install
- What you get
- Setup
- Storage space
- Installation
- Connecting Grafana to Prometheus
- Adding Dashboards
How does it work?
- Node Metrics
- Application Metrics
- So how is this configured?
Wrapping up

Requirements

Kubernetes cluster, preferably AWS EKS
Helm

The quick 5-minute install

What you get

Prometheus Server
Prometheus Node Exporter
Prometheus Alert Manager
Prometheus Push Gateway
Kube State Metrics

Setup

Storage space

Before we begin, it is worth mentioning the file storage requirements of Prometheus. Prometheus server will be running with a persistent volume(PV) attached, and this volume will be used by the time-series database to store the various metrics it collects in the /data folder. Note that we have set our PV to 100Gi in the following line of values.yaml.

https://github.com/ryanoolala/recipes/blob/8a732de67f309a58a45dec2d29218dfb01383f9b/metrics/prometheus/5min/k8s/values.yaml#L765
## Prometheus server data Persistent Volume size
    ##
    size: 100Gi

This will create a 100Gib EBS volume attached to our prometheus-server. So how big of a disk do we need to provide? There are a few factors involved so generally, it will be difficult to calculate the right size for the current cluster without first knowing how many applications are hosted, and also to account for growth in the number of nodes/applications.

Prometheus also has a default data retention period of 15 days, this is to prevent the amount of data from growing indefinitely and can help us keep the data size in check, as it will delete metrics data older than 15 days.

In Prometheus docs, they suggest calculating using this formula, with 1-2 bytes_per_sample

# https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

This is difficult for me to calculate even with an existing setup, so if this is your first time setting up I can imagine it being even more so. So as a guide, I'll share with you my current setup and disk usage, so you can gauge how much of disk space you want to provision.

In my cluster, I am running

20 EC2 nodes
~700 pods
Default scrape intervals
15 day retention

My current disk usage is ~70G.

If the price of 100GiB of storage is acceptable for you, in my region it is about USD12/month. I think it is a good starting point and you can save the time and effort on calculating for storage provisioning and just start with this.

Note that I'm running Prometheus 2.x, that has an improved storage layer over Prometheus 1, and has shown to have reduced the storage usage and thus the lower need of disk space, see blog

With this out of the way, let us get our Prometheus application started.

Installation

$ helm install prometheus stable/prometheus -f https://github.com/ryanoolala/recipes/blob/master/metrics/prometheus/5min/k8s/values.yaml --create-namespace --namespace prometheus

Verify that all the prometheus pods are running

kubectl get pod -n prometheus
NAME                                             READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-78b5c64fd5-ch7hb         2/2     Running   0          67m
prometheus-kube-state-metrics-685dccc6d8-h88dv   1/1     Running   0          67m
prometheus-node-exporter-8xw2r                   1/1     Running   0          67m
prometheus-node-exporter-l5pck                   1/1     Running   0          67m
prometheus-pushgateway-567987c9fd-5mbdn          1/1     Running   0          67m
prometheus-server-7cd7d486cb-c24lm               2/2     Running   0          67m

Connecting Grafana to Prometheus

To access Grafana UI, run kubectl port-forward svc/grafana -n grafana 8080:80, go to http://localhost:8080 and log in with the admin user, if you need the credentials, see the previous post on instructions.

Go to the datasource section under the settings wheel

and click "Add data source"

If you've followed my steps, your Prometheus setup will create a service named prometheus-server in the prometheus namespace. Since Grafana and Prometheus are hosted in the same cluster, we can simply use the assigned internal A record to let Grafana discover prometheus.

Under the URL textbox, enter http://prometheus-server.prometheus.svc.cluster.local:80. This is the DNS A record of prometheus that will be resolvable for any pod in the cluster, including our Grafana pod. Your setting should look like this.

Click "Save & Test" and Grafana will tell you that the data source is working.

Adding Dashboards

Now that Prometheus is setup, and has started to collect metrics, we can start visualizing the data. Here are a few dashboards to get you started.

https://grafana.com/grafana/dashboards/315
https://grafana.com/grafana/dashboards/1860
https://grafana.com/grafana/dashboards/11530

Mouse over the "+" icon and select "Import", paste the dashboard ID into the textbox and click "Load"

Select our Prometheus datasource we added in the previous step into the drop-down selection

You will have a dashboard that looks like this

You may have noticed the "N/A" in a few of the dashboard panels, this is a common problem in various dashboards, due to incompatible versions of Prometheus/Kubernetes with changes in metric labels, etc.
We will have to edit the panel and debug the queries to fix them. If there are too many errors, I will suggest finding another dashboard until you find one that works and fits your needs.

How does it work?

You may have wondered how all these metrics are available, even though you've simply deployed it, without configuring anything other than disk space. To understand the architecture of Prometheus, check out their documentation. I've attached an architecture diagram from the docs here for reference.

We will keep it simple by only focusing on how we are retrieving node and application(running pods) metrics. Remember at the start of the article, we noted down the various Prometheus applications you will get for following this guide.

Prometheus-server pod will be pulling metrics via Http endpoints, most typically the /metrics endpoint from various sources.

Node Metrics

When we installed prometheus, there is a prometheus-node-exporter daemonset that is created. This ensures that every node in the cluster will have one pod of node-exporter, which is responsible for retrieving node metrics and exposing them to its /metrics endpoint.

Application Metrics

prometheus-server will discover services through the Kubernetes API, to find pods with specific annotations. As part of the configuration of the application deployments, you will usually see the following annotations in various other applications.

metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "4000"
    prometheus.io/scrape: "true

These are what prometheus-server will look out for, to scrape for metrics from the pods.

So how is this configured?

Prometheus will load its scraping configuration from a file called prometheus.yml which is a configmap mounted into prometheus-server pod. During our installation using the helm chart, this file is configurable inside the values.yaml, see the source code at values.yaml#L1167. The scrape targets are configured in various jobs and you will see several jobs configured by default, each catering to a specific configuration of how and when to scrape.

An example for our application metrics is found at

#https://github.com/ryanoolala/recipes/blob/8a732de67f309a58a45dec2d29218dfb01383f9b/metrics/prometheus/5min/k8s/values.yaml#L1444
 - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
          - role: pod

        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name

This is what configures prometheus-server to scrape pods with the annotations we talked about earlier.

Wrapping up

With this, you will have a functioning metric collector and dashboards to help kick start your observability journey in metrics. The Prometheus we've set up in this guide will be able to provide a healthy set up in most systems.

However, there is one limitation again to take note of and that is this Prometheus is not set up for High Availability(HA)

As this uses an Elastic Block Store(EBS) volume, as we have explained in the previous post, it will not allow us to scale-out the prometheus-server to provided better service uptime, if the prometheus pod restarts, possibly to due Out-of-memory(OOMKilled) or unhealthy nodes, and if you have alerts set up using metrics, this can be an annoying problem as you will lose your metrics for the time being, and blind to the current situation.

The solution to this problem is something I have yet to deploy myself, and when I do, I will write the part 3 of this series, but if you are interested in having a go at it, check out thanos.

Hope that this has been simple and easy enough to follow and if you have a Kubernetes cluster, even if it is not on AWS, this Prometheus setup is still relevant and be deployed in any system, with a StorageDriver configured to automatically create persistent volumes in your infrastructure.

Getting started with deploying grafana and cloudwatch metric dashboards

Ryan — Wed, 02 Sep 2020 06:10:46 +0000

The Pillar of Metrics

Metrics is one of the key components in observability which is increasingly more important as we adopt more distributed application architectures, monitoring the health of our applications becomes difficult to manage if we don't have an aggregation system in place. If you are just starting on your observability journey, and find justifying for paid SaAS services such as datadog or splunk a tough barrier to overcome, you can easily start by first using open source solutions that can give you a better grasp of how metric collection works, and create dashboards to provide some insights into your current system.

In this post, we will be going through some quick recipes that help deploy Grafana onto an AWS Elastic Kubernetes Service(EKS) cluster with minimal effort, with dashboards created by the community. So let us get started.

Requirements
The quick 5-minute build
- What you get
- Setup
- Cloudwatch IAM Role
- Grafana
  - Installing Grafana
- Show me the UI!
- Whats next?
The 10-minute build
- What you get
- Setup
- Postgres RDS
- Grafana
Logging in
Wrapping up

Requirements

Kubernetes cluster, preferably AWS EKS
Helm
Terraform

The quick 5-minute build

What you get

Grafana instance
Cloudwatch metrics
Cloudwatch dashboards to monitor AWS services(EBS, EC2, etc.)

Setup

ryanoolala / recipes

A collection of recipes for setting up observability toolings

recipes

A collection of recipes for setting up resources in AWS and EKS

This is my attempt of trying to introduce observability tools to people and providing a recipe for them to add them into their infrastructure as easily as possible, as such you may find that most of these setups may be too simple for your production needs(e.g HA consideration, maintenance processes), and if I am able to think of ways to make these better and able to simplify into recipes, I will update this repository, as a recipe guide for myself in my future setups.

Requirements

This repository assumes you already have the following tools installed and required IAM permissions(preferable an admin) to use with terraform

terraform >= v0.12.29
terragrunt >= v0.23.6
kubectl >= 1.18
helm >= 3.3.0

Note

This is not a free tier compatible setup and any costs incurred will be bared by you and you…

View on GitHub

Clone the recipe repository from github.com/ryanoolala/recipes, which I will be using to reference the setup throughout this post

Cloudwatch IAM Role

We first create an IAM role with permissions to get metrics from cloudwatch, and to speed things up we'll be using terraform to provision the role and in my case, I'll be making use of terragrunt, but you can easily copy the inputs and into a terraform module variable input instead.

module "cloudwatch-iam" {
  source = "git::https://gitlab.com/govtechsingapore/gdsace/terraform-modules/grafana-cloudwatch-iam?ref=1.0.0"
  allow_role_arn = arn:aws:iam::{{ACCOUNT_ID}}:role/ryan20200826021839068100000001
  name = "ryan"
}

The grafana cloudwatch iam module takes in a EKS ARN role, this is because we want our Grafana application running on the node, to be able to assume this cloudwatch role, and be authorized to pull metrics from AWS APIs. This provides a terraform output of
grafana_role_arn = arn:aws:iam::{{ACCOUNT_ID}}:role/grafana-cloudwatch-role-ryan

Grafana

Here is where it gets interesting, we will be deploying Grafana using helm 3. Make sure you have your kubectl context set to the cluster you want to host this service on, and that it also belongs to the same AWS account which we just created the IAM role.

We create a datasource.yaml file with the following values, be sure to replace assumeRoleArn with your output from above.

# file://datasource.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Cloudwatch
        type: cloudwatch
        isDefault: true
        jsonData:
          authType: arn
          assumeRoleArn: "arn:aws:iam::{{ACCOUNT_ID}}:role/grafana-cloudwatch-role-ryan"
          defaultRegion: "ap-southeast-1"
          customMetricsNamespaces: ""
    version: 1
    # <bool> allow users to edit datasources from the UI.
    editable: true

This will allow grafana to start with a cloudwatch datasource that is set to use assumeRoleArn for retrieving cloudwatch metrics.

Installing Grafana

$ helm install grafana stable/grafana -f https://github.com/ryanoolala/recipes/blob/master/metrics/grafana/5min/k8s/grafana/values.yaml -f datasource.yaml --create-namespace --namespace grafana

or if you have cloned the repository, place datasource.yaml into ./metrics/grafana/5min/k8s/grafana and run

$ cd ./metrics/grafana/5min/k8s/grafana && make install.datasource

In a few moments, you will have a grafana running

$ kubectl get pod -n grafana
NAME                          READY   STATUS     RESTARTS   AGE
grafana-5c58b66f46-9dt2h      2/2     Running    0          84s

and to get access to the dashboard, run

$ kubectl port-forward svc/grafana -n grafana 8080:80
Forwarding from 127.0.0.1:8080 -> 3000
Forwarding from [::1]:8080 -> 3000

Show me the UI!

Navigate to http://localhost:8080 and you will see your Grafana UI

If you are wondering where these dashboards are loaded, I found them on grafana's dashboard site, picked a few of them, and loaded them by configuring values.yaml

# https://github.com/ryanoolala/recipes/blob/master/metrics/grafana/5min/k8s/grafana/values.yaml#L364

dashboards:
  default:
    aws-ec2:
      url: https://grafana.com/api/dashboards/617/revisions/4/download
    aws-ebs:
      url: https://grafana.com/api/dashboards/11268/revisions/2/download
    aws-cloudwatch-logs:
      url: https://grafana.com/api/dashboards/11266/revisions/1/download
    aws-rds:
      url: https://grafana.com/api/dashboards/11264/revisions/2/download
    aws-api-gateway:
      url: https://grafana.com/api/dashboards/1516/revisions/10/download
    aws-route-53:
      url: https://grafana.com/api/dashboards/11154/revisions/4/download
    aws-ses:
      url: https://grafana.com/api/dashboards/1519/revisions/4/download
    aws-sqs:
      url: https://grafana.com/api/dashboards/584/revisions/5/download

Whats next?

There are some limitations to what we have just deployed, while the UI allows to edit and even add new dashboards, the changes we make are not persistent, since we did not provide any persistent store for this setup. Let's make it better!

The 10-minute build

To save our changes, there are several ways to do so, the easiest probably being attaching a block store(EBS) volume to the instance, and have settings stored on the disk. However as EBS is not a ReadWriteMany storage driver, we cannot scale-out our Grafana instance across availability zones and different EKS nodes. The next easiest solution, in my opinion, will be to make use of AWS Relational Database Service(RDS), which is fully managed, with automatic backups and High Availability(HA), as our persistence layer for Grafana.

What you get

HA Grafana with persistence

Setup

Postgres RDS

We will be using postgres in this example, although Grafana supports MySQL and sqlite3 as well. In order to not digress, I will omit the setup instructions for the database, if you will like to know how I used terraform to deploy the instance, you may read up more in my README.

If you are not familiar with terraform, this might get slightly complicated, thus I will suggest that you create the postgres using the AWS console which will be much easier and faster, to keep it under the 10min effort required for this.

Grafana

Now that we have a postgres database setup, we will create a kubernetes secret object to contain the credentials needed for connecting to it.

$ cd ./metrics/grafana/10min/k8s/grafana
$ make secret
Removing old grafana-db-connection...
secret "grafana-db-connection" deleted
Postgres Host?: 
mydbhost.com
Postgres Username?: 
myuser
Postgres Password? (keys will not show up in the terminal): 
Attempting to create secret 'grafana-db-connection'...
secret/grafana-db-connection created

This secret grafana-db-connection will be used in our values.yaml and we will also set the environment GF_DATABASE_TYPE to postgres.

# https://github.com/ryanoolala/recipes/blob/cf7839e9e919735c72fee77450d891f8ee13ef17/metrics/grafana/10min/k8s/grafana/values.yaml#L268
## Extra environment variables that will be pass onto deployment pods
env:
  GF_DATABASE_TYPE: "postgres"

# https://github.com/ryanoolala/recipes/blob/cf7839e9e919735c72fee77450d891f8ee13ef17/metrics/grafana/10min/k8s/grafana/values.yaml#L282
envFromSecret: "grafana-db-connection"

With these changes done, we can upgrade our current deployed Grafana using helm upgrade grafana stable/grafana -f values.yaml --namespace grafana, or if you are starting from a fresh setup,

$ helm install grafana stable/grafana -f https://github.com/ryanoolala/recipes/blob/master/metrics/grafana/10min/k8s/grafana/values.yaml --create-namespace --namespace grafana

or if you have cloned the repository, run

$ cd ./metrics/grafana/10min/k8s/grafana && make install

This new Grafana will allow you to make changes to the system, add datasources and dashboards, and save these changes in the database so you don't have to worry about your instance restarting and having to start all over again.

Logging in

To make edits, you have to login using the admin user, the default username and password can be set during installation, by modifying the following in values.yaml

# Administrator credentials when not using an existing secret (see below)
adminUser: admin
adminPassword: strongpassword

After Grafana has been started, we can change our password on the UI instead, and the new password will be stored in database for future login sessions.

Wrapping up

Hopefully, this gave you an idea of how you can make use of the grafana helm chart and configure it to display cloudwatch metric dashboards.

In the next part, I will share more about deploying prometheus, which will provide us with more insights within the kubernetes cluster, including CPU/RAM usage of the ec2 nodes, as well as pods. These pieces of information will help us better understand our deployed applications.

DEV Community: Ryan

Getting started with grafana and prometheus for kubernetes metrics

Table of contents

Requirements

The quick 5-minute install

What you get

Setup

Storage space

Installation

Connecting Grafana to Prometheus

Adding Dashboards

How does it work?

Node Metrics

Application Metrics

So how is this configured?

Wrapping up

Getting started with deploying grafana and cloudwatch metric dashboards

The Pillar of Metrics

Table of contents

Requirements

The quick 5-minute build

What you get

Setup

ryanoolala / recipes

A collection of recipes for setting up observability toolings

recipes

Requirements

Cloudwatch IAM Role

Grafana

Installing Grafana

Show me the UI!

Whats next?

The 10-minute build

What you get

Setup

Postgres RDS

Grafana

Logging in

Wrapping up