Setting Up Monitoring on Amazon EKS: Deploying Prometheus and Grafana

Monitoring is a critical aspect of managing Kubernetes workloads. This blog guides you through setting up Prometheus and Grafana on an Amazon EKS cluster to monitor cluster resources and application performance effectively. We will also address potential challenges like storage provisioning and showcase the seamless integration between Prometheus and Grafana.

Step 1: Create an Amazon EKS Cluster

We assume you are starting with an Amazon EKS cluster. Here's how you can create one using eksctl.

EKS Cluster Configuration File

Create a file named cluster.yml with the following content:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: basic-cluster
  region: us-west-2

nodeGroups:
  - name: ng-1
    instanceType: m5a.large
    desiredCapacity: 2
    volumeSize: 30
    ssh:
      allow: false
    iam:
      withAddonPolicies:
        ebs: true
        cloudWatch: true
  - name: ng-2
    instanceType: m5a.large
    desiredCapacity: 2
    volumeSize: 30

iam:
  withOIDC: true`

Run the following command to create the cluster:

eksctl create cluster -f cluster.yml

Step 2: Install the AWS EBS CSI Driver Add-On

Prometheus requires persistent storage for metrics data. To dynamically provision Amazon EBS volumes, install the AWS EBS CSI Driver add-on.

eksctl create addon \
  --name aws-ebs-csi-driver \
  --cluster basic-cluster \
  --region us-west-2

Why the AWS EBS CSI Driver is Required?
The EBS CSI Driver enables Kubernetes to manage EBS volumes dynamically. Prometheus uses PersistentVolumeClaims (PVCs) to request storage, and without this driver, the PVCs cannot bind to an EBS volume, leaving the Prometheus pods in a Pending state.

Step 3: Deploy Prometheus

First, create a dedicated namespace for Prometheus:

kubectl create namespace prometheus

Add the Prometheus Community Helm repository and update Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update prometheus-community

Install Prometheus with persistent storage using Helm:

 helm upgrade -i prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistence.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2" \
    --set server.service.type=LoadBalancer

Ensure all Prometheus pods are running:

kubectl get pods -n prometheus

Prometheus is exposed via a LoadBalancer service. To access the Prometheus server UI:

kubectl get svc -n prometheus

Open your web browser and navigate to EXTERNAL-IP

Step 4: Deploy Grafana

Grafana provides advanced visualization capabilities and integrates seamlessly with Prometheus.

Add the Grafana Helm repository and update Helm:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Deploy Grafana with persistent storage:

helm upgrade -i grafana grafana/grafana \
  --namespace prometheus \
  --set persistence.storageClassName="gp2" \
  --set persistence.enabled=true \
  --set adminPassword="admin123" \
  --set service.type=LoadBalancer

Retrieve the LoadBalancer IP for Grafana:

kubectl get svc -n prometheus | grep grafana

Step 5: Import Dashboard in Grafana

1. Ensure you have imported the 15661 dashboard into Grafana:
2. Navigate to Dashboards > Import.
3. Enter 15661 in the import dialog and select the data source (Prometheus).
4. Save the imported dashboard.