DEV Community: Dennis Tei-Muno

Simulating An Event-Driven Python Shopping App with Kafka on AWS For Real-Time Processing.

Dennis Tei-Muno — Sat, 25 Oct 2025 11:35:46 +0000

Business Use Application/Relevance:

Applications like banking applications, streaming(like netflix), music applications(like spotift), weather application process millions and trillions of livestream real-time data a day. How do they do so? Using Kafka architecture. This project simulates how a shopping app would process numerous shopping orders in a day without lag or breakdown.

Challenges I Faced and How I Solved Them:

My applications kept going offline. I started my applications in tmux terminals that did not die out when I refreshed the page or went away for 10 minutes
My warehouse application not picking up my order: my kafka server configuration for my warehose application was set wrong. Upon fixing it I was able to pickup orders.

How do large companies handle massive traffic for websites without crashing and insane slowness? Enter Apache Kafka. Kafka sits in-betwen applications allowing them in real-time to send and receive real-time traffic and messages. Think of how millions of people get the Amazon packages from order to delivery. How does that happen application-side without incredible lags? Thank kafka. Here I simulate real-time event driven architecture by showing by multiple applications can function with a large user base. An example is how banks server multiple clients through their apps, showing real-time bank account updates, login, fraud check without significant lags. This architecture also shows how companies like Amazon handle millions or purchases based on certain events without crashes.

Project Steps:

Creating Ec2 instance That Will Host Apache Kafka for Me

For simplicity and quickness, I will use the ec2 console to create an ec2 instance with the following specifications: us-east-1, t2.medium, ubuntu image:

I will then launch my ec2 instance and connect to it via ec2 instance connect for SSH for simplicity:

Setting Up Apache Kafka 3.0.0 on Kafka Server For Scala 2.13 on Kafka Server

I will navigate to the website to find the download link for Kafka. You can use any version that would work for my setup: https://kafka.apache.org/downloads

Under binary downloads I will copy the web address for Scala 2.13
I will then execute the command on my terminal:

wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz
tar -xvf kafka_2.13-3.0.0.tgz

I will then install Java(Amazon coretto) as kafka requires java. Follow instructions as per your operating system and CPU architecture. For ubuntu I used the following commands:

# Navigate to https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/downloads-list.html
wget -O - https://apt.corretto.aws/corretto.key | sudo gpg --dearmor -o /usr/share/keyrings/corretto-keyring.gpg && \
echo "deb [signed-by=/usr/share/keyrings/corretto-keyring.gpg] https://apt.corretto.aws stable main" | sudo tee /etc/apt/sources.list.d/corretto.list
sudo apt-get update; sudo apt-get install -y java-1.8.0-amazon-corretto-jdk
java -version

Setting Up Kafka With KRaft Helper:

I will navigate to the directory of the unzipped kafka folder and run the ff commands:

tmux new -s kafka
cd kafka_2.13-3.0.0
bin/kafka-storage.sh random-uuid #this generates a random UUID that will be used in the next step to start the Kafka service

Make sure to copy the random generate uuid:

I will format my kafka storage using the randomly generated uuid:

bin/kafka-storage.sh format -t <randomly-generated-uuid> -c config/kraft/server.properties

I will edit the server properties using the ff:

vim config/kraft/server.properties

I will set the listeners to the following:

I will set the advertised listeners with the IP address of my ec2 instance:

I will then save server.properties text file
In my security group, the following ports are required for Kafka to work: ports 9092(this is the port applications either sending traffic to or consuming data from will hit) and 9093(this is the port kafka brokers will use for intra-cluster communication. :

I have opened ports 9092 and 9093 to the internet in my security group modification. This is bad security practice but since this is a personal project I have opened internet access wide to create ease:

I will now start my kafka service:

bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/kraft/server.properties
# Use CTRL+b, then d to detach session. keeps kafka running in background

If you see something like this and you do not get an error message this means that your kafka server has started:

I will create a new tmux shell for my shopping application. This repo will be for a sample shopping application:

git clone https://github.com/kodekloudhub/kafka.git
cd kafka

In the final_project directory you will find two applications toy-shop and warehouse-ui :

ls final_project/

Configuring Applications to Send Traffic To Kafka

I will navigate to the toy shop application and run the following commands:

cd final_project/toy-shop
tmux new -s shop

If you get the following error from installing the virtual environment install this:

sudo apt install python3.12-venv -y
python3 -m venv .venv #reinstall virtual environment again
source .venv/bin/activate #activate virtual environment

I will install python dependencies for this application in the virtual environment:

pip3 install -r requirements.txt

I will configure my application to connect to Kafka by modifying my application(app.py) to connect to the Kafka server:

vim app.py

Modify the following in app.py and save:

# Kafka producer configuration
conf = {
    'bootstrap.servers': 'localhost:9092',    # Modify to localhost
    'client.id': socket.gethostname()
}

Just for ease I will open port 5000 to the internet. This is not recommended in enterprise settings:

I will start my application with the command:

python3 app.py --host=0.0.0.0 --port=5000
I will detach this shell so that I can keep it running:
# CTRL +b, then d

To view the application, navigate to:

http://<ec2-public-ip-address>:5000

I will add a couple of items to my cart and then at the bottom of my page click "view cart":

After placing my orders I see that logs were generated that showed that kafka received real-time messaging logs about the orders I placed:

 (kafka.server.KafkaConfig)
[2025-10-25 10:00:16,231] INFO [SocketServer listenerType=BROKER, nodeId=1] Starting socket server acceptors and processors (kafka.network.SocketServer)
[2025-10-25 10:00:16,235] INFO [SocketServer listenerType=BROKER, nodeId=1] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2025-10-25 10:00:16,235] INFO [SocketServer listenerType=BROKER, nodeId=1] Started socket server acceptors and processors (kafka.network.SocketServer)
[2025-10-25 10:00:16,236] INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2025-10-25 10:00:16,237] INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser)
[2025-10-25 10:00:16,237] INFO Kafka startTimeMs: 1761386416236 (org.apache.kafka.common.utils.AppInfoParser)
[2025-10-25 10:00:16,237] INFO [Controller 1] The request from broker 1 to unfence has been granted because it has caught up with the last committed metadata offset 9. (org.apache.kafka.controller.BrokerHeartbeatManager)
[2025-10-25 10:00:16,239] INFO Kafka Server started (kafka.server.KafkaRaftServer)
[2025-10-25 10:00:16,242] INFO [Controller 1] Unfenced broker: UnfenceBrokerRecord(id=1, epoch=8) (org.apache.kafka.controller.ClusterControlManager)
[2025-10-25 10:00:16,270] INFO [BrokerLifecycleManager id=1] The broker has been unfenced. Transitioning from RECOVERY to RUNNING. (kafka.server.BrokerLifecycleManager)
[2025-10-25 10:04:01,528] INFO Sent auto-creation request for Set(cartevent) to the active controller. (kafka.server.DefaultAutoTopicCreationManager)
[2025-10-25 10:04:01,544] INFO [Controller 1] createTopics result(s): CreatableTopic(name='cartevent', numPartitions=1, replicationFactor=1, assignments=[], configs=[]): SUCCESS (org.apache.kafka.controller.ReplicationControlManager)
[2025-10-25 10:04:01,545] INFO [Controller 1] Created topic cartevent with topic ID iZpPpV8XR8e7AoLiwX6jGg. (org.apache.kafka.controller.ReplicationControlManager)
[2025-10-25 10:04:01,545] INFO [Controller 1] Created partition cartevent-0 with topic ID iZpPpV8XR8e7AoLiwX6jGg and PartitionRegistration(replicas=[1], isr=[1], removingReplicas=[], addingReplicas=[], leader=1, leaderEpoch=0, partitionEpoch=0). (org.apache.kafka.controller.ReplicationControlManager)
[2025-10-25 10:04:01,581] INFO [ReplicaFetcherManager on broker 1] Removed fetcher for partitions Set(cartevent-0) (kafka.server.ReplicaFetcherManager)
[2025-10-25 10:04:01,598] INFO [LogLoader partition=cartevent-0, dir=/tmp/kraft-combined-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log$)
[2025-10-25 10:04:01,604] INFO Created log for partition cartevent-0 in /tmp/kraft-combined-logs/cartevent-0 with properties {} (kafka.log.LogManager)
[2025-10-25 10:04:01,607] INFO [Partition cartevent-0 broker=1] No checkpointed highwatermark is found for partition cartevent-0 (kafka.cluster.Partition)
[2025-10-25 10:04:01,608] INFO [Partition cartevent-0 broker=1] Log loaded for partition cartevent-0 with initial high watermark 0 (kafka.cluster.Partition)

Viewing Receiving Application: Allowing Warehouse-UI application receive real-time orders for information and processing

I will open up port 5003 on my security group:

I will open another timux terminal:

tmux new -s warehouse
# I will create a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# I will change directory to my warehouse ui-application
cd /root/kafka-project/kafka/final_project/toy-shop
#install python requirements for app:
python -r requirements.txt
# change directory to warehouse-ui application:
cd ../warehouse-ui
vim app.py

Change the bootstrap server in appy.py to localhost:9092:


![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/neh67b6bi47oiaa2w2jp.png)

I will then run my application using :

python3 app.py --port=5003



![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/78fr8ofsvdpv66nn7ydt.png)


- I will visit my warehouse-ui application using:

http://<ec2-public-ip-address:5003


![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/azhpa5y2m7t0169wwppb.png)

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gy6w84yha0xzxtp1u1v0.png)

## Testing my Kafka Setup
- Ensure your kafka server is running:

ss -tulnp | grep 9092





- On my toy-shop UI application I will make a couple of orders and go into my cart and place my order:


![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0m1kltn7jcpbmjhzsl3i.png)

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uf7aow7qab7tn9bav7jf.png)


- Boom! My orders have been picked up in the warehouse picker application:

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/windg22fghp0254joswr.png)


- This shows the power of Kafka where real-time important transaction or events are coordinated between applications. For this example, this is for a shopping application but think weather applications, banking applications and more.

Real-time communication shown here, courtesy kafka in application!

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yphqglkwket536ci3zzz.png)

![ ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d44e6q1xd7at3x1bud2m.png)

Python for DevOps: Python For Provisioning Webserver and Installing Software On It

Dennis Tei-Muno — Sat, 05 Jul 2025 10:33:42 +0000

DevOps Project: EC2 and Apache Deployment with Python & Fabric

This project automates the provisioning and deployment of a basic Apache web application using Python-based DevOps tools. It showcases the integration of AWS (via Boto3), SSH-based automation (via Fabric), and infrastructure scripting to streamline web server deployment.

Project Overview

Goal:

Provision an EC2 instance, upload and install Apache2 using a shell script, and manage infrastructure using Python automation.

Project Structure

run_ec2_instances.py Automates the creation of an EC2 instance using Boto3 with custom tags, security group, subnet, and monitoring options.

import boto3

client = boto3.client("ec2")

def create_ec2_instance(image_id, instance_type, count):
    """
    Launch one or more EC2 instances with the given parameters.
    """
    response = client.run_instances(
        ImageId=image_id,
        InstanceType=instance_type,
        KeyName="keypairforluitccp3",
        MinCount=count,
        MaxCount=count,
        Monitoring={"Enabled": True},
        SecurityGroupIds=["<put-your-here>"],
        SubnetId="<put-yours-here>",
        TagSpecifications=[
            {
                "ResourceType": "instance",       
                "Tags": [
                    {"Key": "environment", "Value": "development"},
                ],
            }
        ],
    )

    return response


if __name__ == "__main__":
    create_ec2_instance("ami-020cba7c55df1f615", "t2.micro", 1)

create_s3_bucket.py Creates a private S3 bucket using Boto3 for optional static file storage or artifact management.

import boto3

client = boto3.client("s3", region_name="us-east-1")   

def create_s3_bucket():
    """
    Create a private S3 bucket named dtm-python-devops2.
    """
    client.create_bucket(
        Bucket="dtm-python-devops2",
        ACL="private"

    )

if __name__ == "__main__":
    create_s3_bucket()

ssh_into_instance.py Uses the Fabric library to:
- SSH into the EC2 instance via PEM key
- Upload the Apache setup script
- Make the script executable and run it with sudo

from fabric import Connection
from subprocess import run
#Connect to the instance using the keypair
c = Connection(
    'ubuntu@23.22.12.129:22', 
    connect_kwargs={
        'key_filename': '/Users/dteimuno/Desktop/keypairforluitccp3.pem'
        },
        )
#Ensure local key is accessible
run(['chmod', '400', '/Users/dteimuno/Desktop/keypairforluitccp3.pem'])
#Transfer the script to the instance
c.put('./apache.sh', '/home/ubuntu/apache.sh')
#Make script executable
c.run('chmod +x /home/ubuntu/apache.sh')
#Run the script on the instance
c.run('sudo /home/ubuntu/apache.sh')

apache.sh A Bash script executed remotely to:
- Install Apache2
- Enable and start the service
- Display the server’s public IP address

#install apache2 on ubuntu
#!/bin/bash
# Update package list
sudo apt update
# Install Apache2
sudo apt install -y apache2
# Enable Apache2 to start on boot
sudo systemctl enable apache2
# Start Apache2 service
sudo systemctl start apache2

Tools & Technologies Used

Python 3
Boto3 – AWS automation SDK
Fabric – SSH and remote task automation
AWS EC2 – Cloud compute provisioning
AWS S3 – Cloud object storage
Apache2 – Web server installed via Bash
Ubuntu – EC2 AMI base OS

🔄 Workflow Summary

Provision EC2 instance with run_ec2_instances.py
Create an S3 bucket using create_s3_bucket.py (optional step)
SSH and deploy Apache using ssh_into_instance.py and apache.sh
Apache is now running and serving web content on the EC2 public IP

Future Enhancements

Automate web app zip upload to S3 and extraction on EC2
Add teardown scripts to delete EC2 and S3 resources
Integrate with CI/CD (e.g., GitHub Actions or Jenkins)
Add error handling and logging

Status

✅ Completed — ready for demonstration or extension.

Kubernetes Deployment with Two-Path Ingress Routing on EKS via Kubernetes Ingress Route Paths

Dennis Tei-Muno — Thu, 22 May 2025 21:13:12 +0000

The GitHub repository for this project can be found here:
GitHub Repo:

Business Use Application/Relevance:

This project shows how to create multi-path traffic directions for two different kubernetes deployments(nginx vs apache). In production workspaces this could be used to provision different paths for getting to different web applications hosted by a company.

Challenges I Faced and How I Solved Them:

Failed nginx & apache endpoints. This happened due to routing complexities. I had to do some debugging and documentation reading to figure this out.

Project Steps:

Creating Terraform Files For EKS Cluster and EBS attached volume:
- I will navigate to GitHub.com and create a new repository:

I will create a dev branch for testing before merging with my main branch:

I will then clone the repository onto my local machine(I used GitHub desktop):

I will then open the repo in Visual Studio Code so that I can make my manifest files there:

I will navigate to the terraform EKS cluster page and copy it to a main.tf file in my repository and make changes:

provider "aws" {
  region = "us-east-1"

}


resource "aws_eks_cluster" "example" {
  name = "dtmcluster"

  access_config {
    authentication_mode = "API"
  }

  role_arn = aws_iam_role.cluster.arn
  version  = "1.32"

  bootstrap_self_managed_addons = false

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose"]
    node_role_arn = aws_iam_role.node.arn
  }

  kubernetes_network_config {
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  vpc_config {
    endpoint_private_access = true
    endpoint_public_access  = true

    subnet_ids = [
        var.subnet_1,
        var.subnet_2,
    ]
  }

  # Ensure that IAM Role permissions are created before and deleted
  # after EKS Cluster handling. Otherwise, EKS will not be able to
  # properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSComputePolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSBlockStoragePolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSLoadBalancingPolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSNetworkingPolicy,
  ]
}

resource "aws_iam_role" "node" {
  name = "eks-auto-node-example"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = ["sts:AssumeRole"]
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKSWorkerNodeMinimalPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEC2ContainerRegistryPullOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role" "cluster" {
  name = "eks-cluster-example"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSComputePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSComputePolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSBlockStoragePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSLoadBalancingPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSNetworkingPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy"
  role       = aws_iam_role.cluster.name
}

For my terraform.tf file I will use the ff:

terraform {
    required_providers {
        aws = {
        source  = "hashicorp/aws"
        version = "6.0.0-beta1"
        }
    }
  backend "s3" {
    bucket = "<your-bucket-name>"
    key    = "backend2.hcl"
    region = "us-east-1"
  }    
    required_version = ">= 1.0.0"
}

To initiate creating my cluster I will via CLI in my directory use the commands:

export AWS_ACCESS_KEY_ID=<my-aws-access-key-id>
export AWS_SECRET_ACCESS_KEY=<my-aws-secret-access-key.
terraform init
terraform fmt
terraform validate
terraform plan
terraform apply -auto-approve

I tried adding the EFS CSI and CoreDNS add-ons with terraform but they took forever so I just added them via the console:

I will configure IAM access entry to my EKS cluster:

To connect to my cluster on my local machine I will use the command:

aws configure
#Then fill out the required constraints
#Then
aws eks update-kubeconfig --name=<cluster-name>
alias k=kubectl

I will then add a managed node group so that ec2 instances can be provisioned for my cluster. I will start by creating an IAM role that will allow my eks to have ec2 instances in the cluster:

I will give myself permission to access the cluster:

(my AWS account # not shown for security reasons)

I will add these policies:

I will on the final page create my access entry with EKS to my cluster:

Service Account with IAM Role For Service Account Creation:
- I will create a namespace called luit by:

k create ns luit

I will enable my IAM OIDC provider since it's not enabled by default:

eksctl utils associate-iam-oidc-provider --cluster=<clusterName> --approve

Nginx Webserver Deployment:
- I will create a manifest file (nginx-deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx-web
  namespace: luit
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-web
  template:
    metadata:
      labels:
        app: nginx-web
    spec:
      containers:
      - name: nginx-web
        image: public.ecr.aws/docker/library/nginx:stable-bookworm
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "250m"

I will then create the deployment with the command:

k create -f nginx-deployment.yaml

Exposing Nginx Deployment with ClusterIP Service:
- I will create a service manifest file that exposes my nginx deployment called service-nginx.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: luit
spec:
  selector:
    app: nginx-web
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

I will then create the service with the command:

k create -f service-nginx.yaml

Apache Deployment Manifest Creation:
- I will create a manifest file(apache-deployment.yaml) containing the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-deployment
  namespace: luit
spec:
  replicas: 3
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
        - name: apache
          image: public.ecr.aws/docker/library/httpd:latest
          ports:
            - containerPort: 80
          command: ["/bin/sh", "-c"]
          args:
            - |
              mkdir -p /usr/local/apache2/htdocs/apache && \
              echo "Welcome to Apache under /apache" > /usr/local/apache2/htdocs/apache/index.html && \
              httpd-foreground
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 5

I will then create the deployment using the command:

k create -f apache-deployment.yaml

Exposing Apache Deployment with ClusterIp Service:
- I will create a file called service-apache.yaml and enter the following within:

apiVersion: v1
kind: Service
metadata:
  name: apache-service
  namespace: luit
spec:
  selector:
    app: apache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

I will then create the service with the command:

k create -f service-apache.yaml

AWS Load Balancer Ingress Controller Installation:
I will reference the documentation here for how to install this via helm:

I will download an IAM Role for my load balancer that allows it to make API calls on my behalf:

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.13.0/docs/install/iam_policy.json

I will create an IAM policy based on the policy I just pulled:

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

create a service account with the following annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-load-balancer-controller
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<ALB_CONTROLLER_ROLE_NAME>

Apply it:

kubectl apply -f aws-load-balancer-service-account.yaml

Next step:

eksctl create iamserviceaccount \
    --cluster=<cluster-name> \
    --namespace=kube-system \
    --name=aws-load-balancer-controller \
    --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
    --override-existing-serviceaccounts \
    --region <aws-region-code> \
    --approve

I will add the eks charts repo:

helm repo add eks https://aws.github.io/eks-charts

I will update the repo:

helm repo update eks

I will install the load balancer controller:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=my-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --version 1.13.0

Kubernetes Ingress Creation with Multi-Path Routing For The Deployments:
Based on documentation I will create an ingress resource manifest file that upon creation will help me with two path-traffic two my pods based on path. The manifest file( ingress.yaml) is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: luit-ingress
  namespace: luit
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/group.name: luit-group
    alb.ingress.kubernetes.io/backend-protocol: HTTP
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx-service
                port:
                  number: 80
          - path: /apache
            pathType: Prefix
            backend:
              service:
                name: apache-service
                port:
                  number: 80

I will create the ingress resource using the command:

k create -f ingress.yaml

To check the ingress resource and how to access my deployments based the Application load balancer DNS, I will use the command:

k -n luit get ingress

To access the deployments based on path I will use via CLI: For nginx:

curl http://k8s-luitgroup-399b9c7f5e-968629939.us-east-1.elb.amazonaws.com/

and for apache:

curl http://k8s-luitgroup-399b9c7f5e-968629939.us-east-1.elb.amazonaws.com/apache

To access by the containers by web, I will use: For apache:

http://<ALB-DNS>/apache/

and for nginx path:

http://<ALB-DNS>/

CleanUp:

I will use the ff to destroy my created resources:

terraform destroy -auto-approve

I will also delete any load balancer create to avoid racking up costs

Sources:

Create IAM Role For Service Account

Apache Web-Server Deployment on Terraform Deployed EKS Cluster with Secure Access, Observability, and Scalable Infrastructure

Dennis Tei-Muno — Sun, 18 May 2025 12:37:49 +0000

Featuring Namespaces, Role-Based Access Control(Cluster-Role and Cluster Rolebinding), Observability(Liveness and Readiness Probe) and a Kubernetes Apache Deployment with 5 replicas. This is my first ever Kubernetes on EKS and I can’t wait to build more!!! The GitHub repo for this project can be found here:
Repo-link
This project was also published on my public portfolio page over on Medium.com at:

Business Use Application/Relevance:

This project shows how a containerized apache web server can be deployed on an EKS provisioned kubernetes cluster and exposed to traffic. The cluster infrastructure is deployed infrastructure-as-code with terraform and then by setting local kubeconfig to the EKS cluster the apache deployment is made.

Challenges I Faced and How I Solved Them:
What service type to use to expose apache deployment. I realized after a lot of trying that a load balancer service would work best for me.
Security groups trouble. Be sure to let security groups include ports you’ve opened in your kubernetes manifests.

Part1: Setup GitHub Repository and Lock It Down

I will create a GitHub repository, and then lock down the repository:

I will create a dev branch:

I will clone the repository and edit with Visual Studio Code:

Part2: Create Kubernetes Cluster Terraform Files and Deploy Infrastructure

In Visual Studio Code I will create the following files for my cluster(I used EKS Auto)
I will then open a terminal and run the commands:

export AWS_ACCESS_KEY_ID=<your-aws-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
terraform init
terraform fmt
terraform validate
terraform plan
terraform apply -auto-approve

My files:

main.tf:


provider "aws" {
  region = "us-east-1"
}


resource "aws_eks_cluster" "example" {
  name = "dtmcluster"

  access_config {
    authentication_mode = "API"
  }

  role_arn = aws_iam_role.cluster.arn
  version  = "1.32"

  bootstrap_self_managed_addons = false

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose"]
    node_role_arn = aws_iam_role.node.arn
  }

  kubernetes_network_config {
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }

  vpc_config {
    endpoint_private_access = true
    endpoint_public_access  = true

    subnet_ids = [
      var.subnet_1,
      var.subnet_2,
      var.subnet_3,
    ]
  }

  # Ensure that IAM Role permissions are created before and deleted
  # after EKS Cluster handling. Otherwise, EKS will not be able to
  # properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSComputePolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSBlockStoragePolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSLoadBalancingPolicy,
    aws_iam_role_policy_attachment.cluster_AmazonEKSNetworkingPolicy,
  ]
}

resource "aws_iam_role" "node" {
  name = "eks-auto-node-example"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = ["sts:AssumeRole"]
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "node_AmazonEKSWorkerNodeMinimalPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodeMinimalPolicy"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role_policy_attachment" "node_AmazonEC2ContainerRegistryPullOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPullOnly"
  role       = aws_iam_role.node.name
}

resource "aws_iam_role" "cluster" {
  name = "eks-cluster-example"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSComputePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSComputePolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSBlockStoragePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSBlockStoragePolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSLoadBalancingPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy"
  role       = aws_iam_role.cluster.name
}

resource "aws_iam_role_policy_attachment" "cluster_AmazonEKSNetworkingPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSNetworkingPolicy"
  role       = aws_iam_role.cluster.name
}

terraform.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.38"
    }
  }
  backend "s3" {
    bucket = "eks-backend-dtm"
    key    = "backend.hcl"
    region = "us-east-1"
  }
}

.gitignore:

variables.tf
.terraform/
*.hcl
variables.tf file whose content I won’t show for security reasons:

Part3: Connect To Cluster; Develop Kubernetes Files Locally and Then Apply Resources On Cluster

I will go my cluster in the AWS console and add the following add-ons to make my cluster even better:

I will navigate to add-ons and move to add some add-ons:

I will move to the next page and click on create recommended role:

I will move to the review page and add these add-ons:

I will then create an IAM Access Entry:

I have developed my kubernetes files: cluster-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: luitrole
rules:
- apiGroups:
  - ""
  resources:
  - '*'
  verbs:
  - '*'

cluster-rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: luitrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: luitrole
subjects:
- kind: ServiceAccount
  name: luitsa
  namespace: luit

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-deployment
  labels:
    app: apache
  namespace: luit
spec:
  replicas: 5
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
      - name: nginx
        image: public.ecr.aws/docker/library/httpd:latest
        command: ["sh", "-c", "touch /tmp/healthy && httpd-foreground"]
        livenessProbe:
          exec:
            command:
            - cat
            - /tmp/healthy
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/healthy
          initialDelaySeconds: 5
          periodSeconds: 5
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80

namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: luit

service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: luitsa
  namespace: luit

service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: luitservice
  namespace: luit
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30080
  selector:
    app: apache
  type: NodePort

I will additionally open up port 30080 on my eks cluster security group:

Connecting To Cluster Via CLI

Connect to AWS CLI using the command:

aws configure
#Enter in information as required

Connect to cluster using command:

aws  eks update-kubeconfig --name=<your-cluster-name>

To check we are truly connected to the cluster I will try the command:

alias k=kubectl
k get all -A

Part4: Test Kubernetes Deployment

I built out my files from earlier. I will create the resources one-by-one in the order that makes sense:

k create -f namespace.yaml
k create -f service-account.yaml
k create -f cluster-role.yaml
k create -f cluster-rolebinding.yaml
#to give my serviceaccount an  IAM  Role that allows access  to  EFS(IAM Role for  Service Account)
sh irsa-oidc.sh
k create -f deployment.yaml
k create -f service.yaml

Our apache deployment should be exposed by a load balancer service(which will create an AWS load balancer). To get the DNS for the load balancer I will use the command:

kubectl get svc -n luit
#Copy the external IP and paste it  into your browser

Cleaning Up
I will push my tracked changes to my github repo and merge with the main branch:

I will navigate to my GitHub repo on the main branch and initiate a merge request:

I will clean up my resources using the command;

terraform destroy -auto-approve