DEV Community

Cover image for Ultimate EKS Baseline Cluster: Part 2 - Storage
Joaquin Menchaca
Joaquin Menchaca

Posted on

Ultimate EKS Baseline Cluster: Part 2 - Storage

This may come as a surprise to some, but the AWS managed Kubernetes service, EKS (Elastic Kubernetes Service), no longer comes with support for storage through Kubernetes persistent volumes, which is required for many applications, such as databases.

You will need to add a (CSI) driver to enable persistent volumes support.

This guide, Part 2 of Ultimate EKS Baseline Cluster series, covers doing just that by installing CSI) called AWS EBS CSI Driver.

0. Prerequisites

These are are some prerequisites and initial steps needed to get started before provisioning a Kubernetes cluster and installing add-ons.

0.1 Knowledge: Systems

image of system concept icons

Basic concepts of systems, such as Linux and the shell (redirection, pipes, process substitution, command substitution, environment variables), as well as virtualization and containers are useful. The concept of a service (daemon) is important.

0.2 Knowledge: Kubernetes

image of Kubernetes concept icons

In Kubernetes, familiarity with service types: ClusterIP, NodePort, LoadBalancer, ExternalName and the ingress resource are important.

Exposure to other types of Kubernetes resource objects used in this guide are helpful: persistentvolumeclaims (pvc), storageclass (sc), pods, deployments, statefulsets (sts), configmaps, serviceaccount (sa) and networkpolicies.

0.3 Tools

image of tool icons

These are the tools used in this article series:

  • AWS CLI [aws] is a tool that interacts with AWS.
  • kubectl client [kubectl] a the tool that can interact with the Kubernetes cluster. This can be installed using adsf tool.
  • eksctl [eksctl] is the tool that can provision EKS cluster as well as supporting VPC network infrastructure.
  • POSIX Shell [sh] such as bash[bash] or zsh [zsh] are used to run the commands.

These tools are highly recommended:

  • adsf [adsf] is a tool that installs versions of popular tools like kubectl.
  • jq [jq] is a tool to query and print JSON data
  • GNU Grep [grep] supports extracting string patterns using extended Regex and PCRE.

0.3 Setup Environment Variables

These environment variables will be used throughout this guide. If opening up a new browser tab, make sure to set the environment variables accordingly.

# variables used to create EKS
export AWS_PROFILE="my-aws-profile" # CHANGEME
export EKS_CLUSTER_NAME="my-unique-cluster-name" # CHANGEME
export EKS_REGION="us-west-2"
export EKS_VERSION="1.26"

# KUBECONFIG variable
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# account id
export ACCOUNT_ID=$(aws sts get-caller-identity \
  --query "Account" \
  --output text
)

# ebs-csi-driver
export ROLE_NAME_ECSI="${EKS_CLUSTER_NAME}_EBS_CSI_DriverRole"
export ACCOUNT_ROLE_ARN_ECSI="arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME_ECSI"
POLICY_NAME_ECSI="AmazonEBSCSIDriverPolicy" # preinstalled by AWS
export POLICY_ARN_ECSI="arn:aws:iam::aws:policy/service-role/$POLICY_NAME_ECSI"

Enter fullscreen mode Exit fullscreen mode

0.4 AWS Setup

There's an assumption that AWS CLI have been setup and configured with a profile. This is required before creating or interacting with an EKS cluster.

You can test access to a configured profile with the following command:

export AWS_PROFILE="<your-profile-goes-here>"
aws sts get-caller-identity
Enter fullscreen mode Exit fullscreen mode

This should show something like the following with values appropriate to your environment, e.g. example output to IAM user named kwisatzhaderach:

{
    "UserId": "AIDAXXXXXXXXXXXXXXXXX",
    "Account": "XXXXXXXXXXXX",
    "Arn": "arn:aws:iam::XXXXXXXXXXXX:user/kwisatzhaderach"
}
Enter fullscreen mode Exit fullscreen mode

0.5 EKS Cluster

This article requires that an EKS cluster has been previously provisioned using eksctl tool.

I wrote a previous article that covered this how to provision EKS with two commands:

0.5.1 Existing EKS Cluster

If you have an existing EKS cluster, but need to configure KUBECONFIG for access, you can run this:

mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# update config pointed to by $KUBECONFIG 
aws eks update-kubeconfig \
  --name $CLUSTER \
  --region $REGION \
  --profile $PROFILE
Enter fullscreen mode Exit fullscreen mode

0.5.2 Create a new EKS Cluster

If you have not setup an EKS cluster, you can set it up with the following commands (~20 minutes process):

mkdir -p $HOME/.kube # conditionally create ~/.kube
# use consistent $KUBECONFIG
export KUBECONFIG=$HOME/.kube/$EKS_REGION.$EKS_CLUSTER_NAME.yaml

# provision EKS + add config to $KUBECONFIG 
eksctl create cluster \
  --version $EKS_VERSION \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME \
  --nodes 3

# setup OIDC provider for least privilege
eksctl utils associate-iam-oidc-provider \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --approve
Enter fullscreen mode Exit fullscreen mode

Details of this were covered in the previous article

0.6 Kubernetes Client Setup

If you use asdf to install kubectl, you can get the latest version with the following:

# install kubectl plugin for asdf
asdf plugin-add kubectl \
  https://github.com/asdf-community/asdf-kubectl.git
asdf install kubectl latest

# fetch latest kubectl 
asdf install kubectl latest
asdf global kubectl latest

# test results of latest kubectl 
kubectl version --short --client 2> /dev/null
Enter fullscreen mode Exit fullscreen mode

This should show something like:

Client Version: v1.27.4
Kustomize Version: v5.0.1
Enter fullscreen mode Exit fullscreen mode

1. AWS EBS CSI driver

Title bar with icons for EBS and CSI

The current versions of EKS starting with 1.23 no longer come with a persistent volume support, so you have to install it on your own. The best method or at least the easiest way to install this, is using EKS add-ons facility. This will install the EBS CSI driver.

Installation of this component will require the following steps:

  1. Create IAM Role (e.g. EBS_CSI_DriverRole) and associate it to Kubernetes service account (i.e. ebs-csi-controller-sa).
  2. Deploy AWS EBS CSI driver using EKS add-ons facility, which also sets up the Kubernetes service account (i.e. ebs-csi-controller-sa) with an association back to the above IAM Role (e.g. EBS_CSI_DriverRole).
  3. Create storage class that uses new the EBS CSI driver

1.1 Setup IAM Role and K8S SA association

The following process will create an IAM Role with permissions to access AWS EBS API. The service account ebs-csi-controller-sa will be created later when installing the driver.

# AWS IAM role bound to a Kubernetes service account
eksctl create iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --attach-policy-arn $POLICY_ARN_ECSI \
  --role-only \
  --role-name $ROLE_NAME_ECSI \
  --approve
Enter fullscreen mode Exit fullscreen mode

This will create a IAM Role, which you can verify with:

aws iam get-role --role-name $ROLE_NAME_ECSI
Enter fullscreen mode Exit fullscreen mode

This should show something like this:

{
    "Role": {
        "Path": "/",
        "RoleName": "mycluster_EBS_CSI_DriverRole",
        "RoleId": "AROAZYKZFDW7YZGW3Q5S7",
        "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/mycluster_EBS_CSI_DriverRole",
        "CreateDate": "2023-07-07T21:19:32+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:aud": "sts.amazonaws.com",
                            "oidc.eks.us-west-2.amazonaws.com/id/6311CF96A267242FD6587B1C29D57F1D:sub": "system:serviceaccount:kube-system:ebs-csi-controller-sa"
                        }
                    }
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600,
        "Tags": [
            {
                "Key": "alpha.eksctl.io/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                "Value": "mycluster"
            },
            {
                "Key": "alpha.eksctl.io/iamserviceaccount-name",
                "Value": "kube-system/ebs-csi-controller-sa"
            },
            {
                "Key": "alpha.eksctl.io/eksctl-version",
                "Value": "0.141.0-dev+5c8318ed5.2023-05-12T11:33:48Z"
            }
        ],
        "RoleLastUsed": {}
    }
}
Enter fullscreen mode Exit fullscreen mode

1.2 Install AWS EBS CSI Drvier

This installation uses the EKS Addons feature to install the component.

# Install Addon
eksctl create addon \
  --name "aws-ebs-csi-driver" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION \
  --service-account-role-arn $ACCOUNT_ROLE_ARN_ECSI \
  --force

# Pause here until driver is active
ACTIVE=""; while [[ -z "$ACTIVE" ]]; do
  if eksctl get addon \
       --name "aws-ebs-csi-driver" \
       --region $EKS_REGION \
       --cluster $EKS_CLUSTER_NAME \
    | tail -1 \
    | awk '{print $3}' \
    | grep -q "ACTIVE"
  then
    ACTIVE="1"
  fi
done
Enter fullscreen mode Exit fullscreen mode

It is important to wait until status changes to ACTIVE before proceeding.

You can inspect the pods created by running the following command:

kubectl get pods \
  --namespace "kube-system" \
  --selector "app.kubernetes.io/name=aws-ebs-csi-driver"
Enter fullscreen mode Exit fullscreen mode

This should show something like:

NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-6d5b7bfd56-bwr5x   6/6     Running   0          2m1s
ebs-csi-controller-6d5b7bfd56-wtxf6   6/6     Running   0          2m2s
ebs-csi-node-hjmf5                    3/3     Running   0          2m2s
ebs-csi-node-tzpgs                    3/3     Running   0          2m2s
Enter fullscreen mode Exit fullscreen mode

You can verify the service account annotations references the IAM Role for the EBS CSI driver.

kubectl get serviceaccount "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --output yaml
Enter fullscreen mode Exit fullscreen mode

This should show something like the following

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/my-unique-cluster-name_EBS_CSI_DriverRole
  creationTimestamp: "2023-07-24T21:12:02Z"
  labels:
    app.kubernetes.io/component: csi-driver
    app.kubernetes.io/managed-by: EKS
    app.kubernetes.io/name: aws-ebs-csi-driver
    app.kubernetes.io/version: 1.21.0
  name: ebs-csi-controller-sa
  namespace: kube-system
  resourceVersion: "1922"
  uid: 1b8ecfdb-ce64-491c-8ced-e08b5519755c
Enter fullscreen mode Exit fullscreen mode

1.2.1 Sidebar: eks-addons vs helm chart?

The CSI driver can be installed using either EKS Addons facility from the previous step, or using the aws-ebs-csi-driver Helm chart. I prefer the EKS addons because of simplicity, but also because it installs an extra snapshot container that doesn't come by default with the helm chart.

I documented the full process using the helm chart in this article:

1.3 Create storage class that uses the EBS CSI driver

In order to use the driver, we will need to create a storage class. You can do so by running the following command:

# create ebs-sc storage class
cat <<EOF | kubectl apply --filename -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF
Enter fullscreen mode Exit fullscreen mode

When completed you can verify the storage class was created with:

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

This should show something like this:

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc          ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   17s
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  30m
Enter fullscreen mode Exit fullscreen mode

1.4 Set new storage class to the default (optional)

This is an optional step. As there’s no functional default storage class, we can set the newly created storage class to be the default with the following commands:

kubectl patch storageclass gp2 --patch \
 '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass ebs-sc --patch \
 '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Enter fullscreen mode Exit fullscreen mode

After this, you can verify the change with:

kubectl get storageclass
Enter fullscreen mode Exit fullscreen mode

This should show something like:

NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ebs-sc (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   31s
gp2                kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  30m
Enter fullscreen mode Exit fullscreen mode

1.5 Testing Persistent Volume

Image for title bar with icon for pvc

In this small test, we deploy a pod that continually writes to the external volume, and a volume claim to allocate storage using the storage class we created earlier.

If this works, the storage will be provisioned in the cloud to create the volume, and then it will be attached to the node and mounted in the pod. If this fails, you will see that the pod will be stuck in pending mode.

# create pod with persistent volume
kubectl create namespace "ebs-test"

# deploy application with mounted volume
cat <<-'EOF' | kubectl apply --namespace "ebs-test" --filename -
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: ubuntu
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
      volumeMounts:
      - name: persistent-storage
        mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: ebs-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 4Gi
EOF
Enter fullscreen mode Exit fullscreen mode

You can test the results of the volume creation with the following command:

kubectl get all,pvc --namespace "ebs-test"
Enter fullscreen mode Exit fullscreen mode

We can also look at the events that took place in this namespace with:

kubectl events --namespace "ebs-test"
Enter fullscreen mode Exit fullscreen mode

This will show something like this:

LAST SEEN           TYPE      REASON                   OBJECT                            MESSAGE
56s                 Warning   FailedScheduling         Pod/app                           0/3 nodes are available: persistentvolumeclaim "ebs-claim" not found. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..
55s                 Normal    WaitForPodScheduled      PersistentVolumeClaim/ebs-claim   waiting for pod app to be scheduled
54s                 Normal    Provisioning             PersistentVolumeClaim/ebs-claim   External provisioner is provisioning volume for claim "ebs-test/ebs-claim"
54s (x2 over 54s)   Normal    ExternalProvisioning     PersistentVolumeClaim/ebs-claim   waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
50s                 Normal    Scheduled                Pod/app                           Successfully assigned ebs-test/app to ip-192-168-22-207.us-west-2.compute.internal
50s                 Normal    ProvisioningSucceeded    PersistentVolumeClaim/ebs-claim   Successfully provisioned volume pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee
48s                 Normal    SuccessfulAttachVolume   Pod/app                           AttachVolume.Attach succeeded for volume "pvc-39b9cf94-8b35-436c-b56d-e2fa587245ee"
44s                 Normal    Pulling                  Pod/app                           Pulling image "ubuntu"
41s                 Normal    Pulled                   Pod/app                           Successfully pulled image "ubuntu" in 2.952872938s (2.952907339s including waiting)
41s                 Normal    Created                  Pod/app                           Created container app
41s                 Normal    Started                  Pod/app                           Started container app
Enter fullscreen mode Exit fullscreen mode

1.6 Delete test application

kubectl delete pod app --namespace "ebs-test"
kubectl delete pvc ebs-claim --namespace "ebs-test"
kubectl delete ns "ebs-test"
Enter fullscreen mode Exit fullscreen mode

2.0 Deleting EKS

When deleting up the EKS cluster, you may want to run through these steps.

2.1 Deleting persistent volume claims

When deleting EKS cluster, if you did not delete persistent volumes, you will have left over unused EBS volumes eating costs.

You should delete all the persistent volume claims, which will delete associate persistent volumes. You can list all of the persistentvolumeclaim resources with the following command:

kubectl get persistentvolumeclaim \
  --all-namespaces | grep -v none
Enter fullscreen mode Exit fullscreen mode

Not that some of these will not be deleted if there is an application running that has a lock to the storage. So you will need to delete the associated application as well.

2.2 Reset Default to original Storage Class

As a precaution, we don’t want to have any resources locked that may prevent deletion of the Kubernetes cluster. Run this command if we changed the defaults earlier.

kubectl patch storageclass ebs-sc --patch \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl patch storageclass gp2 --patch \
  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Enter fullscreen mode Exit fullscreen mode

2.3 IAM Roles

These should be removed when removing Kubernetes with eksctl, but it is a good practice to remove them just in case.

eksctl delete iamserviceaccount \
  --name "ebs-csi-controller-sa" \
  --namespace "kube-system" \
  --cluster $EKS_CLUSTER_NAME \
  --region $EKS_REGION
Enter fullscreen mode Exit fullscreen mode

2.4 Kubernetes cluster

Finally, the Kubernetes cluster itself can be deleted.

eksctl delete cluster \
  --region $EKS_REGION \
  --name $EKS_CLUSTER_NAME
Enter fullscreen mode Exit fullscreen mode

Conclusion

This second article shows add support for storage, called persistent volumes on Kubernetes.

The automation using the eksctl will do the following additional things besides provisioning EKS:

  • setup restricted access to AWS cloud resources using IRSA, which associates KSA with IAM Role using an OIDC provider.
  • install applications using EKS addons facility, specifically the EBS CSI driver.

In future articles, I will cover how to add support for load balancers and network policies, as well as a demo application that demonstrates all of these features.

Top comments (0)