DEV Community: Noura Hussein

Create a simple Ingress NGINX Controller on Amazon EKS Cluster with full example

Noura Hussein — Mon, 31 Oct 2022 15:51:40 +0000

Before talking about the ingress controller lets know first

What is Ingress?

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

Ingress is not actually a type of service. Instead, it is an entry point that sits in front of multiple services in the cluster. It can be defined as a collection of routing rules that govern how external users access services running inside a Kubernetes cluster.

Above image is a simple example where an Ingress sends all its traffic to one Service.

What is an Ingress controller?

An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.

In order for the Ingress resource to work, the cluster must have an ingress controller running.
Only creating an Ingress resource has no effect.

Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster.

Kubernetes as a project supports and maintains AWS, GCE, and nginx ingress controllers.

In this article, I will create an Ingress NGINX Controller and then will create a full example to show you the traffic flow.

I've implemented a terraform module to create an NGINX ingress controller with the full example above in my GitHub repo create-simple-nginx-controller
please have a look at Getting started part in the README file to able to use it.

Prerequisites:

1- Terraform Tool: An IAC tool, use version >= 0.13.1.
2- An existing EKS cluster if you don't know how to create an EKS cluster please have a look at my previous article where I explained to you how to create a simple EKS cluster using terraform.
3- AWS CLI: A command line tool for working with Amazon EKS.
4- kubectl: A command line tool for working with Kubernetes clusters.

Steps:
1- Create EKS cluster.
2- Create an Ingress NGINX controller.
3- Create PODs for app1 and app2.
4- Create Services for app1 and app2.
5- Create an Ingress resource.
6- Run the code.
7- Try to connect to your PODs using the LoadBalancer URL.
8- Clean Up.

1- Create EKS cluster
I'll use the terraform module that I've created before
that module will create a VPC containing 2 public subnets with its public route tables, internet gateway attached to the VPC, and EKS cluster on it with two worker nodes.

please have a look at Getting started part in the README to see how to use it.



module "public_eks_cluster" {
  source         = "git::https://github.com/Noura98Houssien/simple-EKS-cluster.git?ref=v0.0.1"
  vpc_name       = "my-VPC1"
  cluster_name   = "my-EKS1"
  desired_size   = 2
  max_size       = 2
  min_size       = 1
  instance_types = ["t3.medium"]
}

2- Create Ingress NGINX controller

In nginx-ingress-controller.tf file I installed the Nginx controller using helm chart nginx-ingress-controller from the repository https://charts.bitnami.com/bitnami via helm release resource in Terraform.

helm is a package manager that helps you define, install, and upgrade applications running on Kubernetes.



resource "helm_release" "nginx-ingress-controller" {
  name       = "nginx-ingress-controller"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "nginx-ingress-controller"


  set {
    name  = "service.type"
    value = "LoadBalancer"
  }

}

The helm chart will create in a default namespace 2 replicas, 2 deployments, 2 pods, and 2 services for the Nginx ingress controller we will use that service with the type of Load balancer, and that LB will listen on ports 80 and 443 to expose our application from the internet.

3- Create PODs for app1 and app2
In k8s-pods.tf file I created two pods one for app1 and another one for app2
here is an example of creating a pod for app1 and the same will be for app2 but with different values.

In the metadata section we define the pod name my-app1 and give it the label app with the value app1
and that label we will use to select a pod while creating a service.

In the spec section we define the container required parameters like container name "my-app1", and an image name "hashicorp/http-echo" this image will display the text message as in args parameter "Hello from my app 1".



resource "kubernetes_pod_v1" "app1" {
  metadata {
    name = "my-app1"
    labels = {
      "app" = "app1"
    }
  }

  spec {
    container {
      image = "hashicorp/http-echo"
      name  = "my-app1"
      args  = ["-text=Hello from my app 1"]
    }
  }
}

4- Create Services for app1 and app2
In k8s-services.tf file I created two services one for my-app1 pod and another one for my-app2 pod
that we created in the previous step.

Here is an example of creating a service for the "my-app1" pod and the same will be for the "my-app2" pod but with different values.

In the metadata section define the service name "my-app1-service" and in the spec section select the pod that should be behind that service by using the same label we created in the pod resource.
the kubernetes_pod_v1.app1.metadata.0.labels.app should give us the value of the label app in the "my-app1" pod which is "app1".and also we should define which port the service will listen to.
the 5678 is the default port for the hashicorp image.



resource "kubernetes_service_v1" "app1_service" {
  metadata {
    name = "my-app1-service"
  }
  spec {
    selector = {
      app = kubernetes_pod_v1.app1.metadata.0.labels.app
    }
    port {
      port = 5678
    }
  }
}

5- Create Ingress resource
In k8s-ingress.tf file I created an Ingress resource where we will define a collection of routing rules that govern how external users access services running inside a Kubernetes cluster.



resource "kubernetes_ingress_v1" "ingress" {
  wait_for_load_balancer = true
  metadata {
    name = "simple-fanout-ingress"
  }

  spec {
    ingress_class_name = "nginx"

    rule {
      http {
        path {
          backend {
            service {
              name = "my-app1-service"
              port {
                number = 5678
              }
            }
          }

          path = "/app1"
        }

        path {
          backend {
            service {
              name = "my-app2-service"
              port {
                number = 5678
              }
            }
          }

          path = "/app2"
        }
      }
    }

  }
}

Here we have 2 paths, for the /app1 path the backend will be "my-app1-service" which means if the user writes "/app1" at the end of the Loadbalancer URL he should see the hello message from app1

and for the /app2 path the backend will be "my-app2-service"
that means if the user writes "/app2" at the end of the Loadbalancer URL he should see the hello message from app2

Let's run the code to get the Load Balancer URL -that comes after creating the ingress controller- and test those paths

6- Run the code

git clone https://github.com/Noura98Houssien/simple-nginx-ingress-controller.git
cd simple-nginx-ingress-controller/examples
terraform init
terraform plan
terraform apply



Plan: 23 to add, 0 to change, 0 to destroy.
Outputs:
k8s_service_ingress_elb = "aa593a2f34eb34e7daf843d81ba15480-956643453.us-east-1.elb.amazonaws.com"

7- Try to connect to your PODs using the LoadBalancer URL

after applying the code you get the output k8s_service_ingress_elb
or you can get it from the services running in the cluster

To see the services you need first get into your cluster
aws eks update-kubeconfig --region region-code --name my-cluster

that will update your kubectl config file
then write that command to see the services created in the default namespace kubectl get svc



nginx-ingress-controller                   LoadBalancer   172.20.184.7     aa593a2f34eb34e7daf843d81ba15480-956643453.us-east-1.elb.amazonaws.com   80:30047/TCP,443:31165/TCP   13m
nginx-ingress-controller-default-backend   ClusterIP      172.20.231.41    <none>                                                                   80/TCP                       13m

copy and past that URL in your browser and type /app1 at the end of the URL
you should see the page with a message "Hello from my app 1"

Do the same for the /app2 and you should see the page with a message "Hello from my app 2"

8- Clean Up

terraform destroy



Plan: 0 to add, 0 to change, 23 to destroy.
Changes to Outputs:
  - k8s_service_ingress_elb = "aa593a2f34eb34e7daf843d81ba15480-956643453.us-east-1.elb.amazonaws.com" -> null

Summary:

In this article, I explained what ingress and ingress controllers are and how to use them. I created a simple example without diving too deep into complex details to keep it as simple as possible. I hope this has added new value to your knowledge.

Create a simple EKS cluster

Noura Hussein — Fri, 21 Oct 2022 01:15:03 +0000

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.

In this article I'll help you to create a simple, default cluster without expanding into all of the available options. By using infrastructure as a code tool (IAC) HashiCorp Terraform

Prerequisites:
1- Terraform Tool: An IAC tool, use version >= 0.13.1.
2- AWS CLI: A command line tool for working with Amazon EKS.
3- kubectl: A command line tool for working with Kubernetes clusters.

I've created the all next steps in my GitHub repo please follow the Getting started part in the readme to see how to use it.

Steps to create your first EKS cluster:
1- Create VPC and subnets that meet Amazon EKS requirements
2- Create Amazon EKS cluster with required IAM role.
3- Create Amazon EKS node group with required IAM role.
4- Run the code
5- Get into the cluster
6- Clean up

So lets start with the first step..

1-Create VPC and subnets

When you create a cluster, you specify a VPC and at least two subnets that are in different Availability Zones.

VPC requirements and considerations:
in the vpc.tf file I called the VPC module created by terraform with the required parameters to create VPC that meets the requirements of EKS cluster creation.



locals {
  region = data.aws_region.current.name
}
module "vpc" {
  source                  = "git::https://github.com/terraform-aws-modules/terraform-aws-vpc.git?ref=v3.16.0"
  name                    = var.vpc_name
  cidr                    = var.vpc_cidr
  azs                     = ["${local.region}a", "${local.region}b"]
  public_subnets          = cidrsubnets(var.vpc_cidr, 1, 1)
  enable_dns_hostnames    = true
  enable_dns_support      = true
  map_public_ip_on_launch = true

  tags = {
    Name = var.vpc_name
  }
  public_subnet_tags = {
    Name                                        = "public subnet"
    "kubernetes.io/role/elb"                    = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  }


}

The VPC must have a sufficient number of IP addresses available for the cluster, any nodes, and other Kubernetes resources that you want to create.
The VPC must have DNS hostname and DNS resolution support. Otherwise, nodes can't register to your cluster.
The VPC might require VPC endpoints using AWS PrivateLink. we need this in cases if we want to create a private cluster but for simplicity, I'll create a public cluster so there is no need for this now. However, AWS recommends creating a private cluster for the production workload.read more about public and private cluster

Subnets requirements and considerations:

The subnets must have enough available IP addresses to deploy all of your nodes and Kubernetes resources to. AWS recommend at least 16 IP addresses.
The subnets can be a public or private in our case will create public subnets.
The public subnet must auto-assign IPv4 because we will deploy the nodes on it.
If you want to deploy load balancers to a subnet, the subnet must have the following tag:



Key                                 Value
kubernetes.io/role/elb                  1

If you use Version 2.1.1 or earlier of the AWS Load Balancer Controller requires the following tag:



Key                                 Value
kubernetes.io/cluster/my-cluster    shared

at the end of this step, we will have vpc created with 2 public subnets in a different availability zone, public route table and internet gateway.

2-Create Amazon EKS cluster with required IAM role

Kubernetes clusters managed by Amazon EKS make calls to other AWS services on your behalf to manage the resources that you use with the service. Before you can create Amazon EKS clusters, you must create an IAM role with the following IAM policies: "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy" "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
Create a default cluster (public) and specify the subnet ids to be the public subnet.

and that all in eks-cluster.tf



resource "aws_iam_role" "eksClusterRole" {
  name = "eksClusterRole"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.eksClusterRole.name
}

resource "aws_iam_role_policy_attachment" "AmazonEKSVPCResourceController" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
  role       = aws_iam_role.eksClusterRole.name
}

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.eksClusterRole.arn

  vpc_config {
    subnet_ids = module.vpc.public_subnets
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
  # Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.AmazonEKSVPCResourceController,
  ]
}

3- Create Amazon EKS node group with required IAM role

The Amazon EKS node kubelet daemon makes calls to AWS APIs on your behalf. Nodes receive permissions for these API calls through an IAM instance profile and associated policies. Before you can launch nodes and register them into a cluster, you must create an IAM role for those nodes to use when they are launched,So you must create an IAM role with the following IAM policies: "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy" "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
Create a node group in the public subnet and with it's configration(max, min,and desired size) and instance type.

and all that in eks-node-group.tf



resource "aws_iam_role" "eksWorkerNodeRole" {
  name = "eksWorkerNodeRole"

  assume_role_policy = jsonencode({
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.eksWorkerNodeRole.name
}

resource "aws_iam_role_policy_attachment" "AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.eksWorkerNodeRole.name
}

resource "aws_iam_role_policy_attachment" "AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.eksWorkerNodeRole.name
}

resource "aws_eks_node_group" "this" {
  cluster_name    = var.cluster_name
  node_group_name = "${var.cluster_name}-nodeGroup"
  node_role_arn   = aws_iam_role.eksWorkerNodeRole.arn
  subnet_ids      = module.vpc.public_subnets
  instance_types  = var.instance_types
  scaling_config {
    desired_size = var.desired_size
    max_size     = var.max_size
    min_size     = var.min_size
  }

  update_config {
    max_unavailable = 1
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
  # Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
  depends_on = [
    aws_eks_cluster.this,
    aws_iam_role_policy_attachment.AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.AmazonEC2ContainerRegistryReadOnly,
  ]
}

4- Run the code

git clone https://github.com/Noura98Houssien/simple-EKS-cluster.git
cd simple-EKS-cluster
create a file with name terraform.tfvars and copy past the values as in terraform.tfvars.tmpl in example folder.



vpc_name       = "my-VPC1"
vpc_cidr       = "10.0.0.0/16"
cluster_name   = "my-EKS1"
desired_size   = 2
max_size       = 2
min_size       = 1
instance_types = ["t3.medium"]

terraform init
terraform plan
terraform apply as you can see there are 17 resource will be added. and there are some outputs will known after apply. It takes around 10 mins to create. after the creation your environment will be like that

5- Get into the cluster

Update kubectl config file by using that command.

aws eks update-kubeconfig --region region-code --name my-cluster
The example output is as follows.



Added new context arn:aws:eks:region-code:111122223333:cluster/my-cluster to /home/username/.kube/config

Confirm communication with your cluster by running the following command. kubectl get svc The example output is as follows.



NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   15m

Confirm the worker nodes created successfully by running the following command. kubectl get nodes The example output is as follow.



NAME                          STATUS   ROLES    AGE   VERSION
ip-10-0-171-51.ec2.internal   Ready    <none>   12m   v1.23.9-eks-ba74326
ip-10-0-94-160.ec2.internal   Ready    <none>   12m   v1.23.9-eks-ba74326

6- Clean up

run that command to remove all resources

terraform destroy
The example output is as follows.



Plan: 0 to add, 0 to change, 17 to destroy.

summary:
In this article, we learned how to create a simple EKS cluster using Terraform, which is a powerful infrastructure as a code tool.
Hopefully, this has added new value to your knowledge.

Our next step is to deploy the application into the cluster and make it accessible through the internet. I have explained the process in detail here.