Getting up and running on AWS EKS Fargate

Introduction

EKS Fargate has been on my "investigate" list in Notion for quite some time now. With some spare time the other weekend I thought I'd do some digging and see if I could get something mega basic but repeatable up and running for if I need to use EKS Fargate for a proof of concept or something similar.

This post walks through how to get a basic application running on EKS Fargate complete with logging in CloudWatch.

Want to follow along?

Alrighty, so before we do anything constructive the repository for all the bits and bobs I'm showing you below can be found here. Some of the code in the snippets has been left out for brevity.

Be wary, AWS is fun but that fun comes at a cost. Following this will cost you some dough so keep an eye on the Billing & Cost Management Dashboard people!

This post assumes you understand the basics of AWS, Kubernetes and Terraform as I won't be going in to deep on these subjects.

I'll admit, my Terraform could be better, this isn't production worthy Terraform code in anyway or form, just me hacking some things together :)

Some of my examples are based off the official Terraform EKS Examples Github repository which can be viewed here.

Let's Get Started

So before we can even think about doing anything we need somewhere for the Fargate nodes to run i.e. Subnets within a VPC....

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 3.0"

  name = local.name
  cidr = "10.0.0.0/16"

  azs             = ["${local.region}a", "${local.region}b", "${local.region}c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true

  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.name}" = "shared"
    "kubernetes.io/role/elb"              = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.name}" = "shared"
    "kubernetes.io/role/internal-elb"     = 1
  }

  tags = local.tags
}

So here we are utilising the VPC Terraform module which allows us to spin up a basic network structure for our EKS Cluster.

Next we can use the EKS Terraform module for the EKS Cluster...

module "eks" {
  source                          = "terraform-aws-modules/eks/aws"
  version                         = "~> 18.0"

  cluster_name                    = local.name
  cluster_version                 = local.cluster_version
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets
}

resource "null_resource" "kubectl" {
  depends_on = [module.eks]
  provisioner "local-exec" {
    command = "aws eks --region ${local.region} update-kubeconfig --name ${local.name}"
  }
}

Utilising the EKS Terraform Module means we can get a EKS Cluster up and running super quick without having to get bogged down in Terraform code.

In the above code snippet we are also using the null_resource to be able to run a command which updates the local kubeconfig so we can use kubectl commands against the EKS Cluster as soon as the Terraform apply command has finished.

The outputs and providers file haven't been shown in this post but are available in Github Repository.

Right, now we can go ahead and deploy our VPC and EKS Cluster by running the below in a Terminal...

terraform init
terraform apply

Deploying the EKS Cluster can take some time so feel free to grab a Coffee or alternative beverage whilst it does it's thing.

Once Terraform has finished deploying everything you should see a success message followed by the ARN of the Cluster, this specific output has been specified in the outputs file...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

cluster_arn = "arn:aws:eks:eu-west-2:012345678901:cluster/ex-eks-fargate"

To verify everything has gone according to plan we can use kubectl to interact with out newly created Cluster...

➜ kubectl get ns
NAME              STATUS   AGE
default           Active   3m
kube-node-lease   Active   3m
kube-public       Active   3m
kube-system       Active   3m

➜ kubectl get pods -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
coredns-679c9bd45b-f7x8b   0/1     Pending   0          3m
coredns-679c9bd45b-zw52k   0/1     Pending   0          3m

Hmm, that's interesting, the CoreDNS Pods are stuck in Pending...let's dig into that.

Look Ma, No Nodes

So for those of you have worked with the EKS Terraform Module before might have realised that the Worker Groups are missing from the EKS Terraform code.

Instead of using Worker Groups we utilise Fargate Profiles instead.

A Fargate Profile is used to determine which pods use Fargate when launched. The beauty of this is that you could have a EKS Cluster which comprises of Self Managed/AWS Managed nodes and Fargate Profiles, using the Fargate Profiles to run a specific workload.

So how does the Fargate Profile work?

The Profile contains selectors, if a Pod Deployment contains a selector that matches those in the Profile, it's deployed to Fargate.

A basic Fargate Profile and IAM configuration is shown here...

resource "aws_eks_fargate_profile" "default" {
  depends_on             = [module.eks]
  cluster_name           = local.name
  fargate_profile_name   = "default"
  pod_execution_role_arn = aws_iam_role.eks_fargate_profile_role.arn
  subnet_ids             = flatten([module.vpc.private_subnets])

  selector {
    namespace = "kube-system"
  }

  selector {
    namespace = "default"
  }
}

resource "aws_iam_role" "eks_fargate_profile_role" {
  name = "eks-fargate-profile-role"

  assume_role_policy = jsonencode({
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "eks-fargate-pods.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "AmazonEKSFargatePodExecutionRolePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy"
  role       = aws_iam_role.eks_fargate_profile_role.name
}

We give the Profile a name, specify an Execution Role and specify the Subnets we want our Pods deployed into amongst other attributes.

The Selector is responsible for selecting Pods which are deployed to the kube-system and default namespace and running them on Fargate Nodes.

On the IAM side of things, we create a IAM Role and attach the AmazonEKSFargatePodExecutionRolePolicy to it so it can download container images from ECR and attach the IAM Role to the Fargate Profile as shown in the above code. More information regarding the IAM side of things can be viewed here.

Right, now we can go ahead and deploy our Fargate Profile and associated IAM configuration by running terraform apply as we did in the last section and verifying that the operation was successful.

But, as you can see below, the CorsDNS Pods are still Pending...

➜ kubectl get pods -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
coredns-679c9bd45b-m948h   0/1     Pending   0          2m
coredns-679c9bd45b-wwlfn   0/1     Pending   0          2m

Lets dig into why that's happening.

So Close, but we have something to "Patch" up

So we have our EKS Cluster, a Fargate Profile which is setup to run Pods in the kube-system namespace...what else do we need to do?

By default, the CoreDNS deployment is configured to run on Amazon EC2 instances, we need to change that.

If we run the describe kubectl command against one of the CoreDNS Pods we'll see something interesting...

➜  eks-fargate kubectl describe pod/coredns-6f6f94db9c-fjhjn -n kube-system
Name:                 coredns-6f6f94db9c-fjhjn
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 <none>
Labels:               eks.amazonaws.com/component=coredns
                      k8s-app=kube-dns
                      pod-template-hash=6f6f94db9c
Annotations:          eks.amazonaws.com/compute-type: ec2
                      kubectl.kubernetes.io/restartedAt: 2022-05-12T15:11:14+01:00
                      kubernetes.io/psp: eks.privileged
Status:               Pending

We can see that there is an eks.amazonaws.com/compute-type annotation with the value of ec2, we need to remove that annotation, but as we have no control over the Deployment files we'll have to run a patch command against the deployment followed by a rollout restart...

kubectl patch deployment coredns \
      --namespace kube-system \
      --type=json -p='[{"op": "remove", "path": "/spec/template/metadata/annotations", "value": "eks.amazonaws.com/compute-type"}]' \

kubectl rollout restart -n kube-system deployment coredns

The rollout restart command is needed to delete and re-create the existing CoreDNS pods so that they are scheduled on Fargate.

To make life slightly easier, we can place the above code in a script that is ran once the Fargate Profile has been created...

resource "null_resource" "fargate_patch_coredns" {
  depends_on = [aws_eks_fargate_profile.default]
  provisioner "local-exec" {
    command = "/bin/bash ./scripts/patch_coredns_deployment.sh"
  }
}

As in previous steps we can run terraform apply and wait for Terraform to do it's thing.

Fargate Finale

It will take around a minute for the CoreDNS deployment to be deployed onto Fargate after Terraform has finished applying...

➜  kubectl get pods -n kube-system
NAME                       READY   STATUS    RESTARTS   AGE
coredns-86dd9db845-m968g   1/1     Running   0          83s
coredns-86dd9db845-pk27d   1/1     Running   0          83s

Happy days! Let's dig a little deeper...

➜  kubectl get nodes
NAME                                               STATUS   ROLES    AGE   VERSION
fargate-ip-10-0-2-184.eu-west-2.compute.internal   Ready    <none>   89s   v1.22.6-eks-7d68063
fargate-ip-10-0-2-36.eu-west-2.compute.internal    Ready    <none>   84s   v1.22.6-eks-7d68063

So we can see that two Fargate nodes have been fired up, one for each CoreDNS Pod. How exciting!

If we dig into the logs of one of the Pods we can see the events that took place...

Events:
  Type     Reason           Age    From               Message
  ----     ------           ----   ----               -------
  Warning  LoggingDisabled  3m19s  fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
  Normal   Scheduled        2m28s  fargate-scheduler  Successfully assigned kube-system/coredns-86dd9db845-m968g to fargate-ip-10-0-3-190.eu-west-2.compute.internal
  Normal   Pulling          2m27s  kubelet            Pulling image "602401143452.dkr.ecr.eu-west-2.amazonaws.com/eks/coredns:v1.8.7-eksbuild.1"
  Normal   Pulled           2m25s  kubelet            Successfully pulled image "602401143452.dkr.ecr.eu-west-2.amazonaws.com/eks/coredns:v1.8.7-eksbuild.1" in 1.992500087s
  Normal   Created          2m25s  kubelet            Created container coredns
  Normal   Started          2m24s  kubelet            Started container coredns

Alright, great job!

Let's move onto to the next step which is getting the logs from the Pods into CloudWatch, so we can get rid of the Disabled logging because aws-logging configmap was not found error event.

What's going on in there?

In order for the logs of our Pods to be transported to CloudWatch, there are two things that need setting up on the Kubernetes side of things:

1. A Namespace
2. A ConfigMap

I won't go into what the two artifacts above look like as they can be viewed here, but rest assured these are also in the Github repository which can be viewed here.

Utilising a null_resource and a shell script the above mentioned files are deployed once the Fargate Profile has been configured...

resource "null_resource" "deploy_logging_artifacts" {
  depends_on = [null_resource.fargate_patch_coredns]
  provisioner "local-exec" {
    command = "/bin/bash ./logging/deploy.sh"
  }
}

One other thing that we need to do is assign a new policy to our Fargate Profile Role to grant permission to communicate with CloudWatch...

resource "aws_iam_role_policy_attachment" "cloudwatch_transport_policy_attachment" {
  policy_arn = aws_iam_policy.cloudwatch_transport_iam_policy.arn
  role       = aws_iam_role.eks_fargate_profile_role.name
}

resource "aws_iam_policy" "cloudwatch_transport_iam_policy" {
  name = "cloudwatch-transport-iam-policy"
  path = "/"
  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [{
      "Effect" : "Allow",
      "Action" : [
        "logs:CreateLogStream",
        "logs:CreateLogGroup",
        "logs:DescribeLogStreams",
        "logs:PutLogEvents",
      ],
      "Resource" : "*"
    }]
  })
}

As in previous steps, we can run terraform apply and wait for Terraform to do it's thing.

As the CoreDNS Pods aren't the most chattiest of things, to verify the logging is working we can just delete a Pod...

kubectl delete pod/coredns-86dd9db845-pk27d -n kube-system

We can then jump into CloudWatch or use the amazing awslogs to view the output...

➜ awslogs get fluent-bit-cloudwatch --watch --aws-region eu-west-2

fluent-bit-cloudwatch from-fluent-bit-kube.var.log.containers.coredns-5679cff8b6-cmmw8_kube-system_coredns-6e7399229b358343d0469c28a9ca3beaf5646d066250e8c2b76c3b1208f54459.log .:53
fluent-bit-cloudwatch from-fluent-bit-kube.var.log.containers.coredns-5679cff8b6-cmmw8_kube-system_coredns-6e7399229b358343d0469c28a9ca3beaf5646d066250e8c2b76c3b1208f54459.log [INFO] plugin/reload: Running configuration MD5 = 47d57903c0f0ba4ee0626a17181e5d94
fluent-bit-cloudwatch from-fluent-bit-kube.var.log.containers.coredns-5679cff8b6-cmmw8_kube-system_coredns-6e7399229b358343d0469c28a9ca3beaf5646d066250e8c2b76c3b1208f54459.log CoreDNS-1.8.7
fluent-bit-cloudwatch from-fluent-bit-kube.var.log.containers.coredns-5679cff8b6-cmmw8_kube-system_coredns-6e7399229b358343d0469c28a9ca3beaf5646d066250e8c2b76c3b1208f54459.log linux/amd64, go1.17.7, a9adfd56

The log group name is set to fluent-bit-cloudwatch, this is defined in the ConfigMap we deployed at the start of this section.

Basic Application Up & Running

The official Kubernetes site has a basic application that we can try out on EKS Fargate, it can be viewed here.

For the purpose of this post I've created a small script which deploys the application once the EKS Fargate Terraform code and scripts have finished running. So when the application is deployed the Fargate Nodes will have been patched and the logging configuration will have been deployed...

#!/bin/bash

cd guestbook-app

kubectl apply -f https://k8s.io/examples/application/guestbook/redis-leader-deployment.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-leader-service.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-follower-deployment.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-follower-service.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml
kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml

As the Guestbook deployment has no namespace specified it will be deploy into the default one, as we specified the default namespace as a selector earlier on, we don't have to make any changes to accommodate the Guestbook.

Right, to verify everything works correctly I've destroyed the existing setup by running terraform destroy then running terraform apply as we did earlier.

When Terraform has finished applying we should be able to verify that the Guestbook has been deployed successfully...

➜ kubectl get pods                         
NAME                             READY   STATUS    RESTARTS   AGE
frontend-85595f5bf9-9nc27        1/1     Running   0          5m19s
frontend-85595f5bf9-hkj5s        1/1     Running   0          5m23s
frontend-85595f5bf9-jnnpt        1/1     Running   0          5m21s
redis-follower-dddfbdcc9-88cr9   1/1     Running   0          5m25s
redis-follower-dddfbdcc9-j95b9   1/1     Running   0          5m24s
redis-leader-fb76b4755-b4zpx     1/1     Running   0          5m27s

Lets connect to the frontend service we created using port-forward...

➜  kubectl port-forward svc/frontend 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

If everything is working correctly, you should be able to navigate to http://localhost:8080 and interact with the Guestbook application.

Wrapping Up

So there we are, we've run through how to get a basic EKS Fargate Cluster up and running, how to schedule Pods on to Fargate and how to get basic logging functioning.

I hope everyone who has read this post has enjoyed it and if anyone has any questions drop me a comment or a tweet!

Cover Photo by Chuttersnap on Unsplash.

I've dropped a few links in the post but there are a few more below which you may find beneficial.

https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html
https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html
https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html
https://github.com/hashicorp/terraform-provider-kubernetes/issues/723
https://github.com/aws-ia/terraform-aws-eks-blueprints/issues/394
https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1286
https://github.com/vmware-tanzu/octant

Stay safe!

Cheers
DK