DEV Community: Santhosh S

Deploy Mutliple NGINX Ingress on EKS

Santhosh S — Thu, 23 Oct 2025 09:55:13 +0000

To deploy multiple NGINX Ingress Controllers on Amazon EKS with separate ingress classes for internal and external traffic, you'll need to:

1. Create two Helm value files (values-internal.yaml and values-external.yaml)

*values-internal.yaml
*


controller:
  ingressClass: nginx-internal
  ingressClassResource:
    name: nginx-internal
    controllerValue: "k8s.io/ingress-nginx-internal"  # Matches your IngressClass spec
    enabled: false  #Prevent Helm from creating or managing the IngressClass
  ingressClassByName: true
  watchIngressWithoutClass: false
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-################4,subnet-######################"

Note: You have to add below annotation to follow the corrrect nginc ingress class and controller.

Apply appropriate annotations to each controller

IngressClass
  ingressClassByName: true
  watchIngressWithoutClass: false
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"`

values-external.yaml

controller:
  ingressClass: nginx-external
  ingressClassResource:
    name: nginx-external
    controllerValue: "k8s.io/ingress-nginx-external"  # Matches your IngressClass spec
    enabled: false  #Prevent Helm from creating or managing the IngressClass
  ingressClassByName: true
  watchIngressWithoutClass: false
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-09d############,subnet-009#########"

2.Define separate ingress classes (nginx-internal and nginx-external)

Create externa-class.yaml

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-external
spec:
  controller: k8s.io/ingress-nginx-external

Create internal-class.yaml

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-internal
spec:
  controller: k8s.io/ingress-nginx-internal

Once created create custom ingress class:

kubectl apply -f external-class.yaml
kubectl apply -f internal-class.yaml

3.Deploy each controller using Helm with its respective values file:

# Add the ingress-nginx repo
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

# Deploy internal ingress
helm install nginx-internal ingress-nginx/ingress-nginx \
  --namespace ingress-internal --create-namespace \
  -f values-internal.yaml

# Deploy external ingress
helm install nginx-external ingress-nginx/ingress-nginx \
  --namespace ingress-external --create-namespace \
  -f values-external.yaml

Conclusion

Deploying separate NGINX Ingress Controllers for internal and external traffic on EKS enhances security, scalability, and traffic management. By defining distinct ingress classes and customizing Helm values, you gain fine-grained control over how services are exposed—whether privately within your VPC or publicly to the internet.

Deploying EFK stack on Kubernetes cluster using Helm Charts

Santhosh S — Tue, 30 Sep 2025 10:20:03 +0000

Prerequisites

A running Kubernetes cluster (e.g., Minikube, EKS, GKE).
kubectl CLI installed and configured.
Helm installed on your local machine.
Sufficient cluster resources (memory and CPU) for Elasticsearch
Your applications are deployed on K8s using Helm charts, if not, then I am confident you can figure that out, as we would be focussing on a solution using Helm charts

Step 1: Create a Storage Class for Elasticsearch

To set up the necessary storage class, you can create one using the following YAML configuration:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer

Step 2: Installing the AWS EBS CSI Driver using Helm Chart

To proceed with the installation, follow these steps:

helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver

helm repo update

helm install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver

Step 3: Installing Elasticsearch with Helm Chart

Now is the time to install the Helm chart for Elasticsearch, but first, let’s create a namespace for our logging stack:

$ kubectl create namespace efk

We create a dedicated namespace called efk to keep our Elasticsearch and other logging components organized within the Kubernetes cluster.

Next, add the Elasticsearch repository to your Helm:

helm repo add elastic https://helm.elastic.co

This command adds the Elastic Helm chart repository, allowing you to access and install Elasticsearch with Helm.

Now, you can install Elasticsearch using the following Helm chart:

helm install elasticsearch \
 --set service.type=LoadBalancer \
 --set volumeClaimTemplate.storageClassName=ebs-sc \
 --set persistence.labels.enabled=true elastic/elasticsearch -n efk

After the installation, you can retrieve the username and password for Elasticsearch using the following commands:

kubectl get secrets --namespace=efk elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d

kubectl get secrets --namespace=efk elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d

These commands fetch the credentials required to access Elasticsearch securely. The username and password are stored in a Kubernetes secret called elasticsearch-master-credentials within the efk namespace. The base64 -d part is used to decode the base64-encoded values for human-readable access.

you can obtain the LoadBalancer IP for the service and test Elasticsearch using the following commands:

kubectl get svc -n efk

Look for the service with the name associated with Elasticsearch, and under the “EXTERNAL-IP” column, you should see the LoadBalancer IP. It might be pending at first but will eventually get an IP assigned.

http://<LoadBalancer-IP>:9200
When you access this URL, your browser should prompt you for the username and password. Enter the credentials you obtained earlier, and you should be able to access Elasticsearch securely over HTTPS.

Kibana Installation

Step 1: Installing Kibana with LoadBalancer Service Type

To set up Kibana and make it accessible through a LoadBalancer service type, you can use the following Helm installation command:

The elasticsearch repo also provides kibana so we just need to install it by the following command:

helm install kibana --set service.type=LoadBalancer elastic/kibana -n efk

watch all pods in the efk namespace:

kubectl get pods --namespace=efk

Retrieve the Kibana service account token:

kubectl get secrets --namespace=efk kibana-kibana-es-token -ojsonpath=’{.data.token}’ | base64 -d

you can obtain the LoadBalancer IP for the service and open the Kibana UI using the following commands:

kubectl get svc -n efk

After obtaining the LoadBalancer IP, you can open Kibana in your browser. Kibana typically runs on port 5601. Open your web browser and enter the following URL, replacing with the actual LoadBalancer IP:

http://<LoadBalancer-IP>:5601

Use the username and password you obtained for Elasticsearch above.

After that, you should see the following dashboard:

Press enter or click to view image in full size

Fluent Bit Installation:

To set up Fluent Bit for log processing, follow these steps:

Add the Fluent Bit Helm Repository:
You need to add the Fluent Bit Helm repository to access the necessary Helm charts.
Run the following command to add the repository:

helm repo add fluent https://fluent.github.io/helm-charts

Configure Fluent Bit:

Before installing the Helm chart for Fluent Bit, it’s essential to configure the Fluent Bit to correctly access Elasticsearch, which may be part of your EFK (Elasticsearch, Fluent Bit, Kibana) stack. To do this, you need to set up a configuration file. Here’s how to obtain the values file for Fluent Bit and save it in YAML format:

helm show values fluent/fluent-bit > fluentbit-values.yaml

This command fetches the default configuration values for Fluent Bit and saves them in a file named fluentbit-values.yaml. You can modify this file to customize Fluent Bit's settings for your specific Elasticsearch setup and logging needs. After making your changes, you can install Fluent Bit using Helm with your customized configuration.

And update the following changes in the fluentbit-values.yaml file:

1.modify fluentbit-values.yaml elastic search user name and password
Ex:

    `HTTP_User elastic`
    `HTTP_Passwd e7uafnP9WARJEaZX`

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On


  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Index fluent-bit
        Type  _doc
        Host elasticsearch-master
        Port 9200
        HTTP_User elastic
        HTTP_Passwd e7uafnP9WARJEaZX
        tls On
        tls.verify Off
        Logstash_Format On
        Logstash_Prefix logstash
        Retry_Limit False
        Suppress_Type_Name On

    [OUTPUT]
        Name es
        Match host.*
        Index fluent-bit
        Type  _doc
        Host elasticsearch-master
        Port 9200
        HTTP_User elastic
        HTTP_Passwd e7uafnP9WARJEaZX
        tls On
        tls.verify Off
        Logstash_Format On
        Logstash_Prefix node
        Retry_Limit False
        Suppress_Type_Name On

......
......
......

We can use a Lua script and a simple Lua filter for adding the index field in the log event itself and then use that field in the output plugin in the Logstash_Prefix_Key

.........
.........
.........
luaScripts:
  setIndex.lua: |
    function set_index(tag, timestamp, record)
        index = "somePrefix-"
        if record["kubernetes"] ~= nil then
            if record["kubernetes"]["namespace_name"] ~= nil then
                if record["kubernetes"]["container_name"] ~= nil then
                    record["es_index"] = index
                        .. record["kubernetes"]["namespace_name"]
                        .. "-"
                        .. record["kubernetes"]["container_name"]
                    return 1, timestamp, record
                end
                record["es_index"] = index
                    .. record["kubernetes"]["namespace_name"]
                return 1, timestamp, record
            end
        end
        return 1, timestamp, record
    end
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name lua
        Match kube.*
        script /fluent-bit/scripts/setIndex.lua
        call set_index

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Type  _doc
        Host elasticsearch-master
        Port 9200
        HTTP_User elastic
        HTTP_Passwd e7uafnP9WARJEaZX
        tls On
        tls.verify Off
        Logstash_Format On
        Logstash_Prefix logstash
        Retry_Limit False
        Suppress_Type_Name On

    [OUTPUT]
        Name es
        Match host.*
        Type  _doc
        Host elasticsearch-master
        Port 9200
        HTTP_User elastic
        HTTP_Passwd e7uafnP9WARJEaZX
        tls On
        tls.verify Off
        Logstash_Format On
        Logstash_Prefix node
        Retry_Limit False
        Suppress_Type_Name On

......
......
......

Installing helm chart of Fluentbit using custom value file.

helm install fluent-bit fluent/fluent-bit -f fluentbit-values.yaml -n efk

Conclusion

By default, Fluentbit will start gathering container logs from all the pods that are present in the cluster and will push these to the newly deployed ES cluster.

It also listens to the systemd metrics and pushes them to the same ES cluster.

For exploring the logs, first, verify whether the newly created indices are showing on Kibana.

Go to Kibana → Stack Management → Index Management, and under the Indices tab, you should see the 2 newly created indices with the names logstash-yyyy.MM.dd and node-yyyy.MM.dd

For checking your application logs on Kibana, you need to create an Index Pattern for your app (one-time activity).

Index patterns can be created by: Go to Kibana → Stack Management → Data Views → Create data view→ Specify your index pattern and select a timestamp field → save data view to Kibana
Now, you can check your logs by going to Discover → Select your newly created index pattern from the dropdown → Search Logs
now you will see your logs

🛡️Implementing Pod Security Admission in Kubernetes

Santhosh S — Tue, 30 Sep 2025 08:06:22 +0000

This guide covers:

Why PSA replaces PodSecurityPolicies (PSPs)

How PSA works using namespace labels

The three enforcement modes: enforce, audit, and warn

Real-world examples of applying PSA to production and dev environments

“PSA allows cluster administrators to enforce standardized controls without relying on third-party tools or custom configurations.”

Audit Mode:

kubectl label namespace <ns-name>l pod-security.kubernetes.io/enforce=restricted

It does not block pod creation.
It does not show warnings to users.
It does log violations in the Kubernetes audit logs.

Enforce Mode (Blocks non-compliant pods):

kubectl label namespace <your-namespace> \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/enforce-version=latest

Warn Mode (Allows pods but shows warnings):

kubectl label namespace <your-namespace> \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/warn-version=latest

To test this Block mode and warn mode run sample workload with test namespace

apiVersion: v1
kind: Pod
metadata:
  name: insecure-pod
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        runAsUser: 0  # Violates restricted policy
        allowPrivilegeEscalation: true

Note:

This PSA will apply only for new workloads or pods

Conclusion:

Implementing Pod Security Admission (PSA)
Pod Security Admission (PSA) is a powerful built-in Kubernetes feature that replaces the deprecated PodSecurityPolicy (PSP) mechanism. By using namespace-level labels and enforcement modes (enforce, audit, warn), PSA enables cluster administrators to apply consistent, standards-based security controls across workloads—without relying on external tools.

Implementing PSA helps:

Strengthen pod-level security posture

Simplify policy management using native Kubernetes constructs

Gradually roll out restrictions using audit and warn modes before enforcing

Whether you're securing production workloads or sandboxing development environments, PSA offers a flexible and transparent way to enforce best practices. As Kubernetes continues to evolve, PSA is the recommended path forward for pod security.

Karpenter Deployment - on EKS NodeGroup

Santhosh S — Thu, 25 Sep 2025 11:18:21 +0000

Karpenter is an open-source, high-performance Kubernetes cluster autoscaler developed by AWS. Amazon Elastic Kubernetes Service (EKS) provides a powerful and flexible platform for running containerized applications.

Step 1 : Create an IAM Role

Role Name : KarpenterNodeRole-santhosh"

KarpenterNodeRole-santhosh"
Role attached below policy:

AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly
AmazonSSMManagedInstanceCore

Step 2 : Create a Controller IAM Role

Role Name : KarpenterControllereRole-santhosh
Modify Trust relationships
Add OIDC
Account ID


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/6B407ED9BFC9CE681546033D7AD4156A"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.ap-south-1.amazonaws.com/id/6B407ED9BFC9C46033D7AD4156A:aud": "sts.amazonaws.com",
                    "oidc.eks.ap-south-1.amazonaws.com/id/6B407ED9BFC9CE633D7AD4156A:sub": "system:serviceaccount:karpenter:karpenter"
                }
            }
        }
    ]
}

Step 3: Create a KarpenterControllerPolicy-santhosh" policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Karpenter",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ec2:DescribeImages",
        "ec2:RunInstances",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DeleteLaunchTemplate",
        "ec2:CreateTags",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateFleet",
        "ec2:DescribeSpotPriceHistory",
        "pricing:GetProducts"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ConditionalEC2Termination",
      "Effect": "Allow",
      "Action": "ec2:TerminateInstances",
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "ec2:ResourceTag/karpenter.sh/nodepool": "*"
        }
      }
    },
    {
      "Sid": "PassNodeIAMRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::961489959441:role/MF-MS-PRE-PROD-CLUSTER-NodeInstanceRole"
    },
    {
      "Sid": "EKSClusterEndpointLookup",
      "Effect": "Allow",
      "Action": "eks:DescribeCluster",
      "Resource": "arn:aws:eks:ap-south-1:961489959441:cluster/MF-MS-PRE-PROD-CLUSTER"
    },
    {
      "Sid": "AllowScopedInstanceProfileCreationActions",
      "Effect": "Allow",
      "Action": [
        "iam:CreateInstanceProfile"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/kubernetes.io/cluster/MF-MS-PRE-PROD-CLUSTER": "owned",
          "aws:RequestTag/topology.kubernetes.io/region": "ap-south-1"
        },
        "StringLike": {
          "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileTagActions",
      "Effect": "Allow",
      "Action": [
        "iam:TagInstanceProfile"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/MF-MS-PRE-PROD-CLUSTER": "owned",
          "aws:ResourceTag/topology.kubernetes.io/region": "ap-south-1",
          "aws:RequestTag/kubernetes.io/cluster/MF-MS-PRE-PROD-CLUSTER": "owned",
          "aws:RequestTag/topology.kubernetes.io/region": "ap-south-1"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
          "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileActions",
      "Effect": "Allow",
      "Action": [
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:DeleteInstanceProfile"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/MF-MS-PRE-PROD-CLUSTER": "owned",
          "aws:ResourceTag/topology.kubernetes.io/region": "ap-south-1"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowInstanceProfileReadActions",
      "Effect": "Allow",
      "Action": "iam:GetInstanceProfile",
      "Resource": "*"
    }
  ]
}

Step 4 : Add tags to subnets and security groups

aws ec2 create-tags --tags "Key=karpenter.sh/discovery,Value=MF-MS-PRE-PROD-CLUSTER" --resources "sg-06ee27bb7de43cf6e"

Step 5 : Update aws-auth ConfigMap:

We need to allow nodes that are using the node IAM role we just created to join the cluster. To do that we have to modify the aws-auth ConfigMap in the cluster.

kubectl edit configmap aws-auth -n kube-system

you will need to add a section to the mapRoles that looks something like this.

- groups:
  - system:bootstrappers
  - system:nodes
  # - eks:kube-proxy-windows
  rolearn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-santhosh
  username: system:node:{{EC2PrivateDNSName}}

The full aws-auth configmap should have two groups. One for your Karpenter node role and one for your existing node group.

Step 6: Deploy Karpenter

helm install karpenter oci://public.ecr.aws/karpenter/karpenter  --namespace "karpenter" --create-namespace \
    --set "settings.clusterName=santhosh" \
    --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-santhosh" \
    --set controller.resources.requests.cpu=1 \
    --set controller.resources.requests.memory=1Gi \
    --set controller.resources.limits.cpu=1 \
    --set controller.resources.limits.memory=1Gi

output:


[ec2-user@ip-172-31-0-244 ~]$  kubectl get po -A
NAMESPACE     NAME                         READY   STATUS    RESTARTS   AGE
karpenter     karpenter-7d4c9cbd84-vpbfw   1/1     Running   0          29m
karpenter     karpenter-7d4c9cbd84-zjwz4   1/1     Running   0          29m
kube-system   aws-node-889mt               2/2     Running   0          16m
kube-system   aws-node-rnzsk               2/2     Running   0          51m
kube-system   coredns-6c55b85fbb-4cj87     1/1     Running   0          54m
kube-system   coredns-6c55b85fbb-nxwrg     1/1     Running   0          54m
kube-system   kube-proxy-8jmbr             1/1     Running   0          16m

Step 7 : Fetch AMI ID

We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads.

You can retrieve the image ID of the latest recommended Amazon EKS optimized Amazon Linux AMI with the following command

fetch AMI ID using command line:

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.30/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.30/amazon-linux-2/recommended/image_id --query Parameter.Value --output text

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.30/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text

Step 8: Create nodeclass with KMS encryption:


apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: karpenter-core-class
spec:
  amiFamily: AL2023
  role: arn:aws:iam::961489959441:role/Axis-MF-MSIL-PRE-PROD-CLUSTER-NodeInstanceRole
  subnetSelectorTerms:
    - tags:
        kubernetes.io/cluster/Axis-MF-MSIL-PRE-PROD-CLUSTER: owned
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: Axis-MF-MSIL-PRE-PROD-CLUSTER
  amiSelectorTerms:
    - id: ami-060175f7c2d4690ba
  blockDeviceMappings:
    - deviceName: /dev/xvda
    ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
        kmsKeyID: arn:aws:kms:ap-south-1:3668421:key/c2b329ac-bf-47cd-9e054379b

Step 9: Ceate nodepool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: karpenter-core-pool
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-type
          operator: In
          values:
            - m5.xlarge
            - r5.xlarge
            - m5.2xlarge
            - r5.2xlarge
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: karpenter-core-class
      expireAfter: 720h
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m

🚀 Conclusion: Why Karpenter Stands Out ?

Karpenter offers a modern, intelligent approach to Kubernetes node provisioning that significantly enhances cluster performance and operational efficiency. Compared to traditional solutions like Cluster Autoscaler, Karpenter delivers:

⚙️ Improved Node Scheduling Efficiency By dynamically selecting the optimal instance types and sizes, Karpenter ensures your workloads run on the most suitable infrastructure—eliminating the rigidity of predefined node groups.

⚡ Faster Scaling Karpenter reacts to cluster changes in seconds, enabling rapid scaling during traffic spikes or workload surges—far outpacing the slower response times of Cluster Autoscaler.

💰 Cost Optimization With its ability to choose cost-effective instance types and avoid over-provisioning, Karpenter helps organizations reduce cloud spend while maintaining performance.

🛠️ Simpler Configuration Developers benefit from a streamlined setup process, as Karpenter removes the need to manage complex node group configurations and handles provisioning automatically.

Karpenter Deployment Plan – Phased Approach

Santhosh S — Thu, 25 Sep 2025 11:15:00 +0000

Step 1: Karpenter controller role create and attach policy


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Karpenter",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ec2:DescribeImages",
        "ec2:RunInstances",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DeleteLaunchTemplate",
        "ec2:CreateTags",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateFleet",
        "ec2:DescribeSpotPriceHistory",
        "pricing:GetProducts"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ConditionalEC2Termination",
      "Effect": "Allow",
      "Action": "ec2:TerminateInstances",
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "ec2:ResourceTag/karpenter.sh/nodepool": "*"
        }
      }
    },
    {
      "Sid": "PassNodeIAMRole",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::961489959441:role/Axis-MF-MSIL-PRE-PROD-CLUSTER-NodeInstanceRole"
    },
    {
      "Sid": "EKSClusterEndpointLookup",
      "Effect": "Allow",
      "Action": "eks:DescribeCluster",
      "Resource": "arn:aws:eks:ap-south-1:961489959441:cluster/Axis-MF-MSIL-PRE-PROD-CLUSTER"
    },
    {
      "Sid": "AllowScopedInstanceProfileCreationActions",
      "Effect": "Allow",
      "Action": [
        "iam:CreateInstanceProfile"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/kubernetes.io/cluster/Axis-MF-MSIL-PRE-PROD-CLUSTER": "owned",
          "aws:RequestTag/topology.kubernetes.io/region": "ap-south-1"
        },
        "StringLike": {
          "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileTagActions",
      "Effect": "Allow",
      "Action": [
        "iam:TagInstanceProfile"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/Axis-MF-MSIL-PRE-PROD-CLUSTER": "owned",
          "aws:ResourceTag/topology.kubernetes.io/region": "ap-south-1",
          "aws:RequestTag/kubernetes.io/cluster/Axis-MF-MSIL-PRE-PROD-CLUSTER": "owned",
          "aws:RequestTag/topology.kubernetes.io/region": "ap-south-1"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
          "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileActions",
      "Effect": "Allow",
      "Action": [
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:DeleteInstanceProfile"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/Axis-MF-MSIL-PRE-PROD-CLUSTER": "owned",
          "aws:ResourceTag/topology.kubernetes.io/region": "ap-south-1"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowInstanceProfileReadActions",
      "Effect": "Allow",
      "Action": "iam:GetInstanceProfile",
      "Resource": "*"
    }
  ]
}

Step2: Tag security group of cluster:

aws ec2 create-tags --tags "Key=karpenter.sh/discovery,Value=MF-MS-PRE-PROD-CLUSTER" --resources "sg-06ee27bb7de43cf6e"

Jenkins on EKS using EFS

Santhosh S — Thu, 25 Sep 2025 08:08:21 +0000

In this guide, we’ll walk through deploying Jenkins on Amazon EKS with persistent storage backed by AWS EFS using the CSI driver. This setup ensures scalable, durable, and shared storage for Jenkins builds.

Step 1: Install AWS EFS CSI Driver

kubectl apply -k "github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"

Step 2: Set Up AWS Resources

Get VPC ID

aws eks describe-cluster \
  --name hulk-santhosh-cluster \
  --query "cluster.resourcesVpcConfig.vpcId" \
  --output text \
  --region ap-south-1

Get VPC CIDR Range

aws ec2 describe-vpcs \
  --vpc-ids vpc-07937adc3227e4b54 \
  --query "Vpcs[].CidrBlock" \
  --output text \
  --region ap-south-1

Create Security Group

aws ec2 create-security-group \
  --description efs-test-sg \
  --group-name efs-sg \
  --vpc-id vpc-07937adc3227e4b54 \
  --region ap-south-1

Authorize Ingress

aws ec2 authorize-security-group-ingress \ --group-id sg-0be281b6c437376c5 \ --protocol tcp \ --port 2049 \ --cidr 192.168.0.0/16

Step 3: Create EFS File System


aws efs create-file-system \
  --creation-token eks-efs \
  --region ap-south-1

Create Mount Target

aws efs create-mount-target \
  --file-system-id fs-04ec113cee81e30b2 \
  --subnet-id subnet-0a6d27e06ff1e24ed \
  --security-group sg-0be281b6c437376c5

Step 4: Kubernetes Storage Setup

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-04ec113cee81e30b2

PersistentVolumeClaim
`
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: efs-sc
resources:
requests:
storage: 5Gi

Step 5: RBAC for Jenkins


apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
  # Add relevant rules here
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: jenkins

Step 6: Jenkins Service


apiVersion: v1
kind: Service
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  type: ClusterIP
  ports:
    - name: ui
      port: 8080
      targetPort: 8080
    - name: slave
      port: 50000
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: jenkins

Step 7: Jenkins Deployment


`
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
  labels:
    app: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      securityContext:
        fsGroup: 1000
      initContainers:
      - name: volume-permission-fix
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /var/jenkins_home"]
        securityContext:
          runAsUser: 0
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
      containers:
      - name: jenkins
        image: jenkins/jenkins:lts
        ports:
        - containerPort: 8080
        - containerPort: 50000
        volumeMounts:
        - name: jenkins-home
          mountPath: /var/jenkins_home
      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-claim

`

Step 8: Jenkins Credentials

kubectl exec -it <jenkins-pod-name> -n jenkins -- cat /var/jenkins_home/secrets/initialAdminPassword

Step 9: Service Account Token

apiVersion: v1
kind: Secret
metadata:
  name: jenkins-token
  namespace: jenkins
  annotations:
    kubernetes.io/service-account.name: jenkins
type: kubernetes.io/service-account-token

Step 10: Configure Jenkins Kubernetes Cloud
Kubernetes URL: https://kubernetes.default.svc.cluster.local

Namespace: jenkins

Credentials: Service account token

Jenkins URL: http://jenkins.jenkins.svc.cluster.local:8080

Jenkins Tunnel: jenkins.jenkins.svc.cluster.local:50000

Test connection — it should say Connected to Kubernetes.

Pod Template for Jenkins Agents
Name: jenkins-agent

Namespace: jenkins

Labels: jenkins-agent

Usage: Only build jobs with matching label

Container Template
Name: jnlp

Image: jenkins/inbound-agent:latest

Working Dir: /home/jenkins/agent

Allocate pseudo-TTY: