DEV Community: Chinmay Tonape

Routing Traffic on Kubernetes (EKS) with the F5 NGINX Ingress Controller

Chinmay Tonape — Tue, 13 Oct 2026 18:09:24 +0000

If you have run anything on Kubernetes, you have almost certainly used ingress-nginx - the community controller that fronts inbound traffic for roughly half of all clusters. That era is ending. In this post we will look at why you need to move, and then build a complete, working setup on AWS EKS using the F5 NGINX Ingress Controller instead: an EKS cluster provisioned with Terraform, a Network Load Balancer wired up through the AWS Load Balancer Controller, and three flavors of routing (basic, path-based, and host-based) using two sample apps.

The elephant in the room: ingress-nginx is retired

First, the thing you need to know. The community-maintained kubernetes/ingress-nginx project (the controller that uses nginx.ingress.kubernetes.io/ annotations) has been retired. The Kubernetes Steering and Security Response Committees announced the retirement in early 2026, after years of warnings that the project badly needed maintainers. Best-effort maintenance wound down around March 2026, and after that point there are no new releases, no bug fixes, and no security patches - the repository becomes read-only. Existing installs keep running, and the Kubernetes Ingress API itself is unaffected, but running an unpatched, internet-facing proxy is not a position you want to be in.

Source: the official Kubernetes blog, Ingress NGINX Retirement: What You Need to Know. Facts rephrased here for compliance.

That leaves teams with a decision. The two mainstream directions are:

Gateway API - the long-term successor API for L4/L7 routing.
A supported Ingress controller - most naturally the F5 NGINX Ingress Controller (nginx/kubernetes-ingress), which is a different project from the retired community one. It is actively maintained by F5/NGINX Inc., uses the nginx.org/ annotation prefix, and ships its own CRDs (VirtualServer, VirtualServerRoute, TransportServer, Policy) for advanced traffic management.

	Community `ingress-nginx` (retired)	F5 NGINX Ingress Controller
Status	Retired, read-only, no patches	Actively maintained
Maintainer	Kubernetes community	F5 / NGINX Inc.
Helm chart	`kubernetes.github.io/ingress-nginx`	`oci://ghcr.io/nginx/charts/nginx-ingress`
Annotation prefix	`nginx.ingress.kubernetes.io/`	`nginx.org/`
Advanced routing	Annotation-only	`VirtualServer` / `VirtualServerRoute` / `Policy` CRDs

This post takes direction #2.

Architecture

Here is what we are building. A single Network Load Balancer sits at the edge; the F5 NGINX Ingress Controller runs behind it as a reverse proxy and routes requests to two sample Node.js apps that live in a shared namespace (simple-nodejs-app).

                          Internet
                             │
                   DNS (Route 53 alias A)
            app.chinmayto.com / app1.* / app2.*
                             │
                             ▼
              ┌──────────────────────────────┐
              │   AWS Network Load Balancer   │   (provisioned by the
              │        (internet-facing)      │    AWS Load Balancer Controller)
              └──────────────────────────────┘
                             │  target-type: ip
                             ▼
              ┌──────────────────────────────┐
              │  F5 NGINX Ingress Controller  │   namespace: nginx-ingress
              │        (reverse proxy)        │   ingressClassName: nginx
              └──────────────────────────────┘
                   │                      │
        host/path routing rules   host/path routing rules
                   ▼                      ▼
        ┌──────────────────────────────────────────┐
        │          namespace: simple-nodejs-app     │
        │  ┌─────────────────┐  ┌─────────────────┐ │
        │  │  svc + 2 pods   │  │  svc + 2 pods   │ │
        │  │  nodejs-app-1   │  │  nodejs-app-2   │ │
        │  └─────────────────┘  └─────────────────┘ │
        └──────────────────────────────────────────┘

Component breakdown:

Amazon EKS - the cluster, provisioned with Terraform (terraform-aws-modules/eks/aws), with the eks-pod-identity-agent addon enabled.
AWS Load Balancer Controller (LBC) - installed via Helm, granted AWS permissions through EKS Pod Identity (no IRSA/OIDC wiring). The LBC watches the NGINX LoadBalancer Service and provisions an IP-target NLB for it.
F5 NGINX Ingress Controller - installed via Helm into the nginx-ingress namespace. Its Service is annotated so the LBC backs it with an NLB:
- service.beta.kubernetes.io/aws-load-balancer-type: external
- service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
- service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
Route 53 - one alias A record per hostname (app., app1., app2.chinmayto.com), all pointing at the NLB.
Two Node.js apps - nodejs-app-1 and nodejs-app-2, both in the simple-nodejs-app namespace, each a Deployment (2 replicas) fronted by a ClusterIP Service.

Deploy the two apps

Both apps share the simple-nodejs-app namespace. nodejs-app-1 listens on container port 8080, nodejs-app-2 on 8081; both are exposed internally via a ClusterIP Service on port 80.

deploy-simple-nodejs-app-1.yaml:

---
apiVersion: v1
kind: Namespace
metadata:
  name: simple-nodejs-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: simple-nodejs-app
  name: deployment-nodejs-app-1
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nodejs-app-1
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nodejs-app-1
    spec:
      containers:
      - image: public.ecr.aws/n4o6g6h8/simple-nodejs-app-1:latest
        imagePullPolicy: Always
        name: nodejs-app-1
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
  namespace: simple-nodejs-app
  name: service-nodejs-app-1
spec:
  type: ClusterIP
  ports:
    - port: 80
      name: http
      targetPort: 8080
  selector:
    app.kubernetes.io/name: nodejs-app-1

deploy-simple-nodejs-app-2.yaml is the mirror image in the same simple-nodejs-app namespace (service service-nodejs-app-2, container port 8081).

Apply both:

kubectl apply -f k8s-manifests/deploy-simple-nodejs-app-1.yaml
kubectl apply -f k8s-manifests/deploy-simple-nodejs-app-2.yaml

kubectl get svc,pod -n simple-nodejs-app

A note that shapes everything: Ingress is namespaced

A key Kubernetes rule drives the manifest design below: an Ingress can only reference Services in its own namespace. This is why both apps share a single namespace, simple-nodejs-app: it lets one Ingress reference both Services and own all the rules for a given host. (Trying to serve the same host from two Ingresses in different namespaces causes a host collision - the controller accepts one and ignores the other - which is exactly the trap this layout avoids.)

Also note the F5-specific annotation nginx.org/rewrites - it rewrites the request URI before forwarding to the backend. It is the F5 equivalent of the retired controller's nginx.ingress.kubernetes.io/rewrite-target.

1) Basic routing

The simplest case: send everything on a host to one backend. Here all traffic to app.chinmayto.com goes to service-nodejs-app-1.

nginx-ingress-basic-routing.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-nodejs-ingress
  namespace: simple-nodejs-app
  annotations:
    nginx.org/rewrites: "serviceName=service-nodejs-app-1 rewrite=/"
spec:
  ingressClassName: nginx
  rules:
    - host: app.chinmayto.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-nodejs-app-1
                port:
                  number: 80

kubectl apply -f k8s-manifests/nginx-ingress-basic-routing.yaml
kubectl get ingress -n simple-nodejs-app

Access it at http://app.chinmayto.com/.

2) Path-based routing (single Ingress, single namespace)

Route by URL path: /app1 to one app, /app2 to the other - both under the same host app.chinmayto.com. Because both Services live in one namespace, this is a single Ingress with two paths.

There is a catch: the sample apps only serve /, so the matched prefix (/app1, /app2) has to be stripped before the request reaches the backend - otherwise the app replies Cannot GET /app1. With the F5 NGINX Ingress Controller you strip a prefix using two annotations together:

nginx.org/path-regex: "case_sensitive" - treat the path field as a regular expression.
nginx.org/rewrite-target: "/$1" - rewrite the matched URI to /, where $1 is the regex capture group that preserves anything after the prefix.

nginx-ingress-path-based-routing.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-nodejs-ingress
  namespace: simple-nodejs-app
  annotations:
    nginx.org/path-regex: "case_sensitive"
    nginx.org/rewrite-target: "/$1"
spec:
  ingressClassName: nginx
  rules:
    - host: app.chinmayto.com
      http:
        paths:
          - path: /app1(?:/(.*))?
            pathType: ImplementationSpecific
            backend:
              service:
                name: service-nodejs-app-1
                port:
                  number: 80
          - path: /app2(?:/(.*))?
            pathType: ImplementationSpecific
            backend:
              service:
                name: service-nodejs-app-2
                port:
                  number: 80

kubectl apply -f k8s-manifests/nginx-ingress-path-based-routing.yaml
kubectl get ingress -n simple-nodejs-app

Access http://app.chinmayto.com/app1 and http://app.chinmayto.com/app2.

3) Host-based routing (single Ingress, single namespace)

Route by the HTTP Host header: app1.chinmayto.com to one app, app2.chinmayto.com to the other. Since the hosts differ and both Services are in the same namespace, this is a single Ingress with two host rules.

Each host routes / straight to its backend, so no rewrite annotation is needed - the request path passes through unchanged. (Adding nginx.org/rewrites here is unnecessary and, with two services listed, can misroute and produce a 404.)

nginx-ingress-host-based-routing.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-nodejs-ingress
  namespace: simple-nodejs-app
spec:
  ingressClassName: nginx
  rules:
    - host: app1.chinmayto.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-nodejs-app-1
                port:
                  number: 80
    - host: app2.chinmayto.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-nodejs-app-2
                port:
                  number: 80

kubectl apply -f k8s-manifests/nginx-ingress-host-based-routing.yaml
kubectl get ingress -n simple-nodejs-app

Access http://app1.chinmayto.com and http://app2.chinmayto.com.

Testing without DNS

If your Route 53 records are not in place yet, you can still prove the routing works by sending the Host header explicitly to the NLB. If this returns your app, the controller and manifests are correct and the only remaining piece is DNS:

# Grab the NLB hostname
kubectl get svc nginx-ingress-controller -n nginx-ingress

# Hit it with the right Host header
curl -H "Host: app.chinmayto.com" http://<NLB-DNS-NAME>/

A plain request to the raw NLB name returns an nginx 404 - that is expected. It means the request reached the controller but no rule matched the (wrong) Host header. Point your DNS at the NLB and use the real hostname.

Cleanup

First remove the application and routing manifests, then tear down the infrastructure.

# Delete the Ingress + app manifests
kubectl delete -f k8s-manifests/nginx-ingress-basic-routing.yaml
kubectl delete -f k8s-manifests/nginx-ingress-path-based-routing.yaml
kubectl delete -f k8s-manifests/nginx-ingress-host-based-routing.yaml
kubectl delete -f k8s-manifests/deploy-simple-nodejs-app-1.yaml
kubectl delete -f k8s-manifests/deploy-simple-nodejs-app-2.yaml

# Destroy the EKS cluster, controllers, NLB, and Route 53 records
cd infrastructure
terraform destroy

Conclusion

ingress-nginx served the community well, but with its retirement it is time to move. The F5 NGINX Ingress Controller is a drop-in-spirited, actively maintained alternative: you keep the familiar Kubernetes Ingress resource, swap the annotation prefix to nginx.org/, and gain a richer CRD-based API when you need it. In this post we built the whole path end to end on EKS - an IP-target NLB via the AWS Load Balancer Controller, the F5 controller as the edge proxy, and basic, path-based, and host-based routing driven by plain Ingress resources in a single namespace. A key lesson along the way: because an Ingress can only reference Services in its own namespace, same-host routing is cleanest when the backends share a namespace - which is why both apps live in simple-nodejs-app. From here, the natural next steps are TLS termination, authentication, and traffic splitting with the VirtualServer CRD (which is also F5's answer to true cross-namespace routing).

References

Companion GitHub repo: https://github.com/chinmayto/kubernetes-f5-nginx-ingress
Ingress NGINX Retirement - official Kubernetes blog: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/
Ingress NGINX statement from the Steering & Security committees: https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/
F5 NGINX Ingress Controller docs: https://docs.nginx.com/nginx-ingress-controller/

Kubernetes Gateway API on Amazon EKS with Traefik

Chinmay Tonape — Mon, 12 Oct 2026 17:06:13 +0000

Introduction

If you've been running Kubernetes workloads in production, you've almost certainly used an Ingress controller - probably nginx. It's been the de facto standard for routing external HTTP traffic into a cluster for years. But that era is coming to an end.

The nginx Ingress controller has announced end of active support, and the Kubernetes community has responded with a better answer: the Gateway API. In this post we'll walk through what Gateway API is, why it matters, how different controllers implement it, and how to deploy a real Node.js application on Amazon EKS using Traefik as the Gateway controller - all provisioned with Terraform.

What Is the Kubernetes Gateway API?

The Gateway API is a collection of official Kubernetes CRDs (Custom Resource Definitions) designed to replace the Ingress resource. It graduated to GA (v1.0) in October 2023 and has been steadily growing in adoption since.

Where Ingress is a single flat resource with limited expressiveness, Gateway API introduces a layered model with clear separation of concerns:

Resource	Owned by	Purpose
`GatewayClass`	Cluster operator	Defines which controller manages Gateways
`Gateway`	Cluster operator	Configures the actual listener (port, protocol, TLS)
`HTTPRoute`	Application team	Defines routing rules to backend Services
`TCPRoute`, `TLSRoute`, `GRPCRoute`	Application team	Protocol-specific routing

This separation matters in real organizations. Platform teams control the Gateway (and therefore what load balancer gets provisioned), while application teams deploy HTTPRoutes into their own namespaces without needing cluster-wide permissions. That's something the old Ingress spec could never cleanly express.

Why Move Away from nginx Ingress?

The Ingress-nginx project - the community-maintained nginx Ingress controller - announced it is entering maintenance-only mode. New features will not be added, and the recommendation from the Kubernetes SIG-Network team is to migrate to Gateway API. The writing has been on the wall for a while: Ingress is a frozen API spec, and the ecosystem has moved on.

If you're starting a new project today, Gateway API is the right choice. If you have existing Ingress resources, now is the time to plan the migration.

How Gateway API Controllers Work

Gateway API itself is just a specification - a set of CRDs. The actual work of provisioning load balancers and routing traffic is done by a Gateway controller. Any controller can implement the spec; they just need to watch for GatewayClass resources that advertise their controllerName.

Some popular implementations:

Controller	`controllerName`	Notes
Traefik	`traefik.io/gateway-controller`	Cloud-native, fast, excellent DX
Envoy Gateway	`gateway.envoyproxy.io/gatewayclass-controller`	CNCF project, powerful
Istio	`istio.io/gateway-controller`	Service mesh with Gateway API
Cilium	`io.cilium/gateway-controller`	eBPF-based, very high performance
AWS LBC	`eks.amazonaws.com/load-balancer-controller`	AWS-native NLB/ALB provisioning
nginx	`k8s.nginx.org/nginxgateway`	NGINX Gateway Fabric (new project)

The key insight: the controller is pluggable. Your HTTPRoute and Gateway manifests stay the same regardless of which controller you choose. You only change the GatewayClass.

Why Traefik?

For this walkthrough we chose Traefik because:

It's battle-tested in production and has first-class Gateway API support
It installs via a single Helm chart with no IAM roles or AWS-specific setup
It creates its own Service of type LoadBalancer, letting the AWS cloud-controller-manager provision the NLB automatically - no AWS Load Balancer Controller needed
The Traefik dashboard gives you live routing visibility out of the box

Architecture

Here's what we're building:

Internet
    │
    ▼
Route53 A-alias records (app / app1 / app2 .chinmayto.com)
    │   all three aliased to the SAME NLB hostname
    ▼
AWS Network Load Balancer          ← ONE NLB, provisioned automatically by EKS
    │                                 when Traefik Service type=LoadBalancer
    ▼
Traefik Pod (Deployment, 2 replicas, namespace: traefik)
    │   GatewayClass: traefik
    │   Gateway: traefik (namespace: traefik, listener "web" on port 80)
    ▼
HTTPRoute(s) + apps (one isolated namespace per part)
    │   Part_01 → namespace nodejs-app-part1
    │   Part_02 → namespace nodejs-app-part2
    │   Routing differs per part (see "Routing Patterns" below):
    │     Part_01 → by hostname   (app1 / app2)
    │     Part_02 → by path + URL rewrite (/app1/*, /app2/*)
    ▼
ClusterIP Services → Node.js Pods
    service-nodejs-app-1 (port 80 → container 8080)
    service-nodejs-app-2 (port 80 → container 8081)

Only one NLB is created. All routing patterns and both apps sit behind the
single Traefik LoadBalancer Service. The app, app1, and app2 Route53
records all alias to that one NLB; traffic is differentiated inside the cluster
at the Gateway/HTTPRoute layer, not at the load balancer.

Infrastructure components (all Terraform-managed):

VPC - two public + two private subnets across two AZs, single NAT gateway
EKS cluster - managed node group, t3.medium, AL2023, Kubernetes 1.36
EKS Addons - vpc-cni, kube-proxy, coredns, eks-pod-identity-agent
Gateway API CRDs - installed via kubectl apply from the official v1.2.0 release
Traefik - installed via Helm chart v33.x, Gateway API provider enabled
Route53 A record - Terraform reads the NLB hostname from the Traefik Service and creates the alias record automatically

No AWS Load Balancer Controller. No IRSA. No separate IAM policy. Traefik handles the gateway layer; AWS cloud-controller-manager handles the NLB provisioning.

Routing Patterns

This repo ships two Node.js apps and demonstrates two common Gateway API routing
patterns. Each pattern lives in its own folder under k8s-manifests/ and is
self-contained - it includes both app deployments plus the HTTPRoute(s) specific
to that pattern, deployed into its own dedicated namespace so the parts are
fully isolated from one another and can coexist on the same cluster/NLB.

	Pattern	Folder	Namespace	Entry URL(s)	Backend
Part 01	Hostname-Based Routing	`k8s-manifests/Part_01`	`nodejs-app-part1`	`app1.chinmayto.com`, `app2.chinmayto.com`	app-1 / app-2 by Host header
Part 02	URL Rewriting	`k8s-manifests/Part_02`	`nodejs-app-part2`	`app.chinmayto.com/app1/`, `app.chinmayto.com/app2/`	app-1 / app-2, prefix stripped

Both parts target the same Traefik Gateway - the one auto-created by the
Helm chart. The HTTPRoutes attach to it via:

parentRefs:
  - name: traefik        # Gateway auto-created by the Traefik Helm chart
    namespace: traefik   # Traefik deploys it into its own namespace
    sectionName: web     # the named "web" listener (port 80)

Part 01 - Hostname-Based Routing

Two HTTPRoutes, each matching a distinct hostname. The request Host header
decides which backend serves it. Uses the app1/app2 Route53 subdomains.

# app1.chinmayto.com -> service-nodejs-app-1
hostnames: ["app1.chinmayto.com"]
rules:
  - matches:
      - path: { type: PathPrefix, value: / }
    backendRefs:
      - { name: service-nodejs-app-1, port: 80 }
---
# app2.chinmayto.com -> service-nodejs-app-2
hostnames: ["app2.chinmayto.com"]
rules:
  - matches:
      - path: { type: PathPrefix, value: / }
    backendRefs:
      - { name: service-nodejs-app-2, port: 80 }

Part 02 - URL Rewriting

One HTTPRoute on a single hostname matching a path prefix, with a URLRewrite
filter that strips the prefix before forwarding. The backend only ever sees /,
so apps that know nothing about the sub-path keep working behind it. (This replaces
a naive exact-path approach, where the backend would receive /app1 unchanged and
return Cannot GET /app1 because it only serves /.)

hostnames: ["app.chinmayto.com"]
rules:
  - matches:
      - path: { type: PathPrefix, value: /app1 }
    filters:
      - type: URLRewrite
        urlRewrite:
          path: { type: ReplacePrefixMatch, replacePrefixMatch: / }
    backendRefs:
      - { name: service-nodejs-app-1, port: 80 }
  - matches:
      - path: { type: PathPrefix, value: /app2 }
    filters:
      - type: URLRewrite
        urlRewrite:
          path: { type: ReplacePrefixMatch, replacePrefixMatch: / }
    backendRefs:
      - { name: service-nodejs-app-2, port: 80 }

Step-by-Step Deployment

1. Deploy Infrastructure

terraform plan
terraform apply

This single apply does everything:

Creates the VPC, subnets, NAT gateway
Provisions the EKS cluster and node group
Installs EKS addons
Installs Gateway API CRDs (v1.2.0)
Installs Traefik via Helm (with enableGatewayAPI: true)
Waits 90 seconds for the NLB to be provisioned by AWS
Reads the NLB hostname from the Traefik Service
Creates the Route53 A-alias record

At the end of apply you'll see:

Outputs:

nodejs_app_dns_info = {
  "app_url"        = "http://app.chinmayto.com"
  "app1_hostname"  = "app1.chinmayto.com"
  "app1_url"       = "http://app1.chinmayto.com"
  "app2_hostname"  = "app2.chinmayto.com"
  "app2_url"       = "http://app2.chinmayto.com"
  "full_hostname"  = "app.chinmayto.com"
  "hosted_zone_id" = "Z0XXXXXXXXXXXXXXXXX"
  "nlb_hostname"   = "xxxx.elb.us-east-1.amazonaws.com"
  "subdomain"      = "app"
  "domain_name"    = "chinmayto.com"
}

Terraform creates three Route53 A-alias records (app, app1, app2), all
pointing at the single Traefik NLB. Which hostnames/paths actually route depends
on which part's manifests you apply next.

2. Deploy Kubernetes Manifests

# Update kubeconfig (Terraform does this automatically, but run it if needed)
aws eks --region us-east-1 update-kubeconfig --name CT-EKS-Cluster

# Pick ONE routing part and apply everything in its folder.
# Each folder contains both app deployments plus its HTTPRoute(s).

# Part 01 - Hostname-Based Routing
kubectl apply -f k8s-manifests/Part_01/

# Part 02 - URL Rewriting
kubectl apply -f k8s-manifests/Part_02/

Each part deploys into its own namespace (nodejs-app-part1 / nodejs-app-part2),
so applying a part never touches the other part's resources. Part_01 uses distinct
hostnames (app1/app2) and Part_02 uses app, so the two can run side by side
without clashing. To fully remove a part, delete its namespace:
kubectl delete namespace nodejs-app-part2.

4. Verify

# Traefik pod running?
kubectl get pods -n traefik

# GatewayClass accepted?
kubectl get gatewayclass traefik

# Gateway ready? (auto-created by the Traefik Helm chart in the traefik namespace)
kubectl get gateway traefik -n traefik

# HTTPRoute(s) accepted? (use the namespace for the part you deployed)
kubectl get httproute -n nodejs-app-part1   # or -part2
# ...or across all namespaces:
kubectl get httproute -A

# Application responding? (use the URL(s) for the part you deployed)

# Part 01 - Hostname-Based Routing
curl http://app1.chinmayto.com
curl http://app2.chinmayto.com

# Part 02 - URL Rewriting (note the trailing slash; the /app1 prefix is stripped)
curl http://app.chinmayto.com/app1/
curl http://app.chinmayto.com/app2/

Key Terraform Resources

Traefik Helm release (gateway-api.tf)

resource "helm_release" "traefik" {
  name       = "traefik"
  repository = "https://traefik.github.io/charts"
  chart      = "traefik"
  namespace  = kubernetes_namespace_v1.traefik.metadata[0].name
  version    = "33.2.1"

  values = [yamlencode({
    providers = {
      kubernetesGateway = { enabled = true }
      kubernetesIngress = { enabled = false }
    }
    service = {
      type = "LoadBalancer"
      annotations = {
        "service.beta.kubernetes.io/aws-load-balancer-type"   = "nlb"
        "service.beta.kubernetes.io/aws-load-balancer-scheme" = "internet-facing"
      }
    }
    gateway = {
      enabled = true
      name    = "traefik"
    }
  })]
}

Route53 record from Traefik Service (app-dns.tf)

resource "time_sleep" "wait_for_nlb" {
  create_duration = "90s"
  depends_on      = [helm_release.traefik]
}

data "kubernetes_service_v1" "traefik" {
  metadata {
    name      = "traefik"
    namespace = "traefik"
  }
  depends_on = [time_sleep.wait_for_nlb]
}

# app.<domain>  - used by Part_02 (URL rewrite)
resource "aws_route53_record" "nodejs_app" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = "${var.app_subdomain}.${var.domain_name}"
  type    = "A"

  alias {
    name                   = data.kubernetes_service_v1.traefik.status[0].load_balancer[0].ingress[0].hostname
    zone_id                = local.nlb_hosted_zone_id
    evaluate_target_health = true
  }
}

# app1.<domain> / app2.<domain> - used by Part_01 (hostname-based routing).
# Both alias to the SAME NLB; the HTTPRoute hostnames do the differentiation.
resource "aws_route53_record" "nodejs_app_1" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = "${var.app1_subdomain}.${var.domain_name}"
  type    = "A"

  alias {
    name                   = data.kubernetes_service_v1.traefik.status[0].load_balancer[0].ingress[0].hostname
    zone_id                = local.nlb_hosted_zone_id
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "nodejs_app_2" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = "${var.app2_subdomain}.${var.domain_name}"
  type    = "A"

  alias {
    name                   = data.kubernetes_service_v1.traefik.status[0].load_balancer[0].ingress[0].hostname
    zone_id                = local.nlb_hosted_zone_id
    evaluate_target_health = true
  }
}

Gateway (auto-created by the Traefik Helm chart)

There's no hand-written Gateway manifest. The Traefik chart (gateway.enabled = true)
creates a Gateway named traefik in the traefik namespace with a web listener
on port 80. Every HTTPRoute attaches to it via parentRefs:

parentRefs:
  - name: traefik
    namespace: traefik
    sectionName: web

HTTPRoute manifest - Part 01 example (Part_01/gateway-api-nodejs-app.yaml)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: nodejs-app-1-httproute
  namespace: nodejs-app-part1
spec:
  parentRefs:
    - name: traefik
      namespace: traefik
      sectionName: web
  hostnames:
    - "app1.chinmayto.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: service-nodejs-app-1
          port: 80

See the Routing Patterns section for the Part 02 (URL rewrite)
variant.

Cleanup

To tear everything down cleanly - delete the Kubernetes resources first so the NLB is deprovisioned before Terraform tries to destroy the VPC.

# 1. Delete the Kubernetes resources for whichever part(s) you deployed.
#    Deleting a part's folder removes its Namespace, HTTPRoutes, Deployments
#    and Services in one go.
kubectl delete -f k8s-manifests/Part_01/

kubectl delete -f k8s-manifests/Part_02/

# 2. Wait ~60 seconds for the NLB to be fully removed by AWS
sleep 60

# 3. Destroy all Terraform-managed infrastructure
cd infrastructure
terraform destroy

Why the order matters: If you run terraform destroy while the Traefik LoadBalancer Service still exists, AWS will not delete the NLB (it's owned by Kubernetes). The NLB's ENIs will still be attached to the VPC subnets, and Terraform will fail trying to delete the VPC with a dependency error. Always drain the k8s resources first.

Conclusion

The Kubernetes Gateway API is a genuine step forward from the Ingress spec. The separation between GatewayClass, Gateway, and HTTPRoute maps cleanly onto how real platform and application teams work. The nginx Ingress controller served us well for years, but with active development winding down and Gateway API reaching GA, now is the right time to migrate.

Traefik makes this particularly straightforward on EKS - no IAM roles, no IRSA, no separate controller installation just to provision a load balancer. One Helm chart, one helm_release resource in Terraform, and you have a production-ready Gateway API implementation with an NLB and Route53 DNS fully automated.

The full deployment - VPC, EKS, Traefik, NLB, Route53 - is a single terraform apply. That's the kind of operational simplicity that makes a difference at scale.

References

GitHub Repository: chinmayto/terraform-aws-eks-gateway-api
Kubernetes Gateway API - Official Docs
Gateway API - Supported Features by Implementation
Traefik Gateway API Documentation

AWS CDK Constructs Explained: L1, L2, and L3 with Python

Chinmay Tonape — Mon, 16 Mar 2026 17:42:14 +0000

If you've ever looked at two CDK stacks doing the same thing and wondered why one is 200 lines and the other is 20, the answer is almost always which level of construct the author chose. This post breaks down all three levels — L1, L2, and L3 — using a consistent real-world example so you can see exactly what each abstraction buys you.

Background: What Is AWS CDK?

The AWS Cloud Development Kit (CDK) is an open-source framework that lets you define cloud infrastructure using familiar programming languages — Python, TypeScript, Java, and more. Under the hood, CDK synthesizes your code into AWS CloudFormation templates, which are then used to provision resources.

The key advantage over raw CloudFormation YAML is that you get the full power of a programming language: loops, conditionals, type checking, reusable classes, and IDE support. Instead of copying and pasting 300 lines of YAML every time you need a Lambda function, you write a Python class once and reuse it everywhere.

CDK apps are organized around three concepts:

App — the root of your CDK application
Stack — maps to a CloudFormation stack; a unit of deployment
Construct — the building block. Everything in CDK is a construct.

What Are CDK Constructs?

Constructs are the core abstraction in CDK. Every resource you define — a Lambda function, an S3 bucket, an API Gateway — is a construct. They form a tree: your App contains Stacks, Stacks contain constructs, and constructs can contain other constructs.

CDK ships with three levels of constructs, each offering a different trade-off between control and convenience:

Level	Class prefix	What it represents	Boilerplate
L1	`Cfn*`	Raw CloudFormation resource, 1:1 mapping	Maximum
L2	e.g. `Function`, `RestApi`	Curated resource with defaults and helpers	Medium
L3	Custom class or `aws-solutions-constructs`	Opinionated, reusable multi-resource pattern	Minimum

Understanding when to use each level is one of the most practical skills you can develop as a CDK user.

The Example: A Hello World API

To make the comparison concrete, we'll build the same thing three times — once per construct level:

A REST API with a single GET /hello endpoint, backed by a Lambda function, with CloudWatch Logs capturing both API Gateway access logs and Lambda execution logs.

Prerequisites

Install the required packages

pip install -r requirements.txt

L1 — Raw CloudFormation Wrappers

L1 constructs are the lowest level. Every class is prefixed with Cfn (short for CloudFormation) and maps directly to a CloudFormation resource type. If a property exists in CloudFormation, it exists on the L1 construct — nothing more, nothing less.

Working with L1 feels a lot like writing CloudFormation YAML, just in Python. You are responsible for everything: IAM roles, resource dependencies, Lambda invoke permissions, deployment ordering.

When to reach for L1:

A property you need isn't exposed by the L2 construct yet
You're migrating an existing CloudFormation template to CDK
You need precise control over every resource attribute

# Every piece is explicit — IAM role, log group, function, permissions...
lambda_role = iam.CfnRole(
    self, "LambdaRole",
    assume_role_policy_document={
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {"Service": "lambda.amazonaws.com"},
            "Action": "sts:AssumeRole",
        }],
    },
    managed_policy_arns=[
        "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
    ],
)

lambda_log_group = CfnLogGroup(
    self, "LambdaLogGroup",
    log_group_name="/aws/lambda/l1-hello-world",
    retention_in_days=7,
)

lambda_fn = CfnFunction(
    self, "HelloFunction",
    runtime="python3.12",
    handler="index.handler",
    role=lambda_role.attr_arn,
    code=CfnFunction.CodeProperty(zip_file="..."),
    logging_config=CfnFunction.LoggingConfigProperty(
        log_group=lambda_log_group.log_group_name,
        log_format="JSON",
    ),
)
lambda_fn.add_dependency(lambda_log_group)  # you manage ordering manually

# API Gateway — every piece by hand
rest_api = CfnRestApi(self, "HelloApi", name="l1-hello-world-api")
hello_resource = CfnResource(
    self, "HelloResource",
    rest_api_id=rest_api.ref,
    parent_id=rest_api.attr_root_resource_id,
    path_part="hello",
)
# + CfnMethod, CfnDeployment, CfnStage, CfnPermission, CfnAccount...

Pros: Total control. No hidden behavior. Great for auditing exactly what gets deployed.

Cons: Extremely verbose. You own every IAM permission, every add_dependency(), every Fn.sub() for ARN construction. Easy to miss something.

L2 — Curated Constructs with Defaults

L2 constructs are the sweet spot for most CDK users. They wrap L1 constructs with sensible defaults, helper methods, and automatic wiring. CDK generates IAM roles, adds Lambda invoke permissions, links log groups, and creates deployments — all without you asking.

The mental model shifts from "how do I configure this resource" to "what do I want this resource to do."

When to reach for L2:

Day-to-day infrastructure work (this should be your default)
You want secure defaults without writing boilerplate
You want IDE autocomplete and type safety on resource props

# CDK creates the execution role automatically
hello_fn = _lambda.Function(
    self, "HelloFunction",
    runtime=_lambda.Runtime.PYTHON_3_12,
    handler="index.handler",
    code=_lambda.Code.from_inline("..."),
    timeout=Duration.seconds(10),
    log_group=lambda_log_group,   # just pass the log group, CDK links it
)

# LambdaIntegration automatically adds the resource-based policy
api = apigw.RestApi(
    self, "HelloApi",
    deploy_options=apigw.StageOptions(
        access_log_destination=apigw.LogGroupLogDestination(apigw_log_group),
        access_log_format=apigw.AccessLogFormat.json_with_standard_fields(...),
        logging_level=apigw.MethodLoggingLevel.INFO,
    ),
)
hello_resource = api.root.add_resource("hello")
hello_resource.add_method("GET", apigw.LambdaIntegration(hello_fn))

Pros: Dramatically less code. Secure defaults. Typed props with IDE support. Handles the boring stuff.

Cons: Occasionally a property isn't exposed yet. When that happens, you escape to L1 via .node.default_child — a bit awkward but workable.

L3 — Reusable Patterns

L3 constructs are where CDK really shines for teams. They're custom constructs you build by composing L2 constructs into a higher-level, opinionated pattern. The goal is to encode your team's standards — logging, security, tagging, naming conventions — once, and reuse them everywhere.

Think of L3 as your internal infrastructure library. Instead of every developer wiring up Lambda + API Gateway + CloudWatch from scratch, they instantiate your LambdaApiService and get all of that for free.

AWS also ships a set of official L3 patterns via the AWS Solutions Constructs library — pre-built, well-architected combinations like aws-apigateway-lambda or aws-lambda-dynamodb.

When to reach for L3:

You find yourself copy-pasting the same L2 pattern across multiple stacks
You want to enforce team standards (logging, retention, tagging) in one place
You're building a platform that other teams consume

class LambdaApiService(Construct):
    """
    One construct = Lambda + API Gateway + CloudWatch Logs.
    All wiring is internal. Consumers just pass props.
    """
    def __init__(self, scope: Construct, id: str, *, props: LambdaApiServiceProps):
        super().__init__(scope, id)

        # All the L2 wiring happens here, once, for everyone
        lambda_log_group = logs.LogGroup(self, "LambdaLogs", ...)
        self._function = _lambda.Function(self, "Function", log_group=lambda_log_group, ...)

        apigw_log_group = logs.LogGroup(self, "ApiGwLogs", ...)
        self._api = apigw.RestApi(self, "Api", deploy_options=apigw.StageOptions(
            access_log_destination=apigw.LogGroupLogDestination(apigw_log_group), ...
        ))

        resource = self._api.root.add_resource(props.path)
        resource.add_method("GET", apigw.LambdaIntegration(self._function))

        self.endpoint_url = f"{self._api.url}{props.path}"

Using it in a stack becomes trivial:

class L3HelloWorldStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        # One line gets you Lambda + API GW + CloudWatch, fully wired
        service = LambdaApiService(
            self, "HelloService",
            props=LambdaApiServiceProps(
                function_name="l3-hello-world",
                handler_code=HANDLER_CODE,
                api_name="l3-hello-world-api",
                path="hello",
            ),
        )
        CfnOutput(self, "ApiUrl", value=service.endpoint_url)

Pros: Maximum reuse. New services inherit logging, IAM, and API Gateway automatically. Standards enforced at the construct level, not by convention.

Cons: Upfront design investment. Over-abstraction can hide details that matter. Needs versioning if shared across teams.

Side-by-Side Comparison

	L1	L2	L3
Lines of code (this example)	~130	~60	~40 stack + ~60 construct
IAM roles	Manual	Auto-generated	Auto-generated
Lambda permissions	Manual `CfnPermission`	`LambdaIntegration` handles it	Encapsulated
Log group linking	Manual `add_dependency`	`log_group=` prop	Encapsulated
Reusability	None	Low	High
Escape hatch needed	Never	Rarely	Rarely
Best for	Migration / edge cases	Day-to-day work	Platform / shared libraries

Deploying

# Bootstrap your AWS account (once per account/region)
# Run from inside any of the construct folders
cd l2_construct $&& cdk bootstrap
# Deploy the constructs
cd l1_construct && cdk deploy
cd l2_construct && cdk deploy
cd l3_construct && cdk deploy

Test the endpoint:

L1 Construct:

L2 Construct:

L3 Construct:

Check logs in CloudWatch:

Lambda logs: /aws/lambda/l2-hello-world
API Gateway logs: /aws/apigateway/l2-hello-world

Cleanup

To avoid ongoing charges, destroy the stacks when you're done:

cd l1_construct && cdk destroy
cd l2_construct && cdk destroy
cd l3_construct && cdk destroy

CDK will prompt for confirmation before deleting.

Note: The CDK bootstrap stack (CDKToolkit) is not removed by cdk destroy. You can delete it manually from the CloudFormation console if you no longer need CDK in that account/region.

Conclusion

CDK constructs aren't a hierarchy where L3 is always better — they're tools for different situations. L1 gives you the full CloudFormation surface area with no magic. L2 handles the 80% case cleanly with sensible defaults. L3 is where you invest once and multiply the value across your entire team.

A practical rule of thumb: start with L2, drop to L1 when you hit a wall, and promote repeated L2 patterns into L3 constructs when you find yourself copy-pasting. That progression naturally leads to a clean, maintainable infrastructure codebase.

The real power of CDK isn't any single construct level — it's that all three levels compose together seamlessly in the same stack.

References

GitHub repo with all code from this post: https://github.com/chinmayto/aws-cdk-constructs-python
AWS CDK Python API Reference
AWS CDK Construct Library
AWS Solutions Constructs — official L3 patterns
CDK Patterns — community-contributed patterns

Building Scalable Web Application on AWS with CDK in TypeScript

Chinmay Tonape — Tue, 24 Feb 2026 18:45:26 +0000

Introduction

Infrastructure as Code (IaC) has revolutionized how we provision and manage cloud resources. In our previous blog post, we built a highly available web infrastructure using AWS CDK with Python. We explored the fundamentals of CDK, CloudFormation integration, and deployed a production-ready architecture with EC2 instances behind an Application Load Balancer.

In this tutorial, we'll deploy the exact same architecture, but this time using TypeScript as our CDK language. This gives you the opportunity to:

Compare Python and TypeScript approaches to infrastructure as code
Understand how CDK provides a consistent experience across different programming languages
Choose the language that best fits your team's expertise and preferences
See how type safety and IDE support differ between languages

Whether you're a JavaScript/TypeScript developer looking to manage infrastructure in a familiar language, or you're exploring different CDK language options, this guide will show you how TypeScript brings strong typing, excellent IDE support, and a robust ecosystem to your infrastructure code.

Architecture Overview

We'll build a highly available web infrastructure with the following components:

┌─────────────────────────────────────────────────────────────────────┐
│                            Internet                                 │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
                    ┌────────────────────────┐
                    │   Internet Gateway     │
                    └────────────────────────┘
                                 │
┌────────────────────────────────┼────────────────────────────────────┐
│                                │                  VPC               │
│                                ▼                                    │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │          Application Load Balancer (ALB)                    │    │
│  │                  (Public Subnets)                           │    │
│  └──────────────────────┬──────────────────┬───────────────────┘    │
│                         │                  │                        │
│  ┌──────────────────────┼──────────────────┼─────────────────────┐  │
│  │  Availability Zone 1 │                  │ Availability Zone 2 │  │
│  │                      │                  │                     │  │
│  │  ┌───────────────────▼────────────┐  ┌──▼──────────────────┐  │  │
│  │  │   Public Subnet (AZ-1)         │  │ Public Subnet (AZ-2)│  │  │
│  │  │                                │  │                     │  │  │
│  │  │  ┌──────────────────┐          │  │                     │  │  │
│  │  │  │  NAT Gateway     │          │  │                     │  │  │
│  │  │  └────────┬─────────┘          │  │                     │  │  │
│  │  └───────────┼────────────────────┘  └─────────────────────┘  │  │
│  │              │                                                │  │
│  │  ┌───────────▼────────────────┐  ┌────────────────────────┐   │  │
│  │  │   Private Subnet (AZ-1)    │  │  Private Subnet (AZ-2) │   │  │
│  │  │                            │  │                        │   │  │
│  │  │  ┌──────────────────┐      │  │  ┌──────────────────┐  │   │  │
│  │  │  │  EC2 Instance 1  │      │  │  │  EC2 Instance 2  │  │   │  │
│  │  │  │                  │      │  │  │                  │  │   │  │
│  │  │  │                  │      │  │  │                  │  │   │  │
│  │  │  └──────────────────┘      │  │  └──────────────────┘  │   │  │
│  │  │                            │  │                        │   │  │
│  │  └────────────────────────────┘  └────────────────────────┘   │  │
│  │                                                               │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Building the Infrastructure

Step 1: Project Setup

First, ensure you have the prerequisites installed:

# Install Node.js and npm (if not already installed)
# Install AWS CDK CLI globally
npm install -g aws-cdk

# Verify installation
cdk --version

Step 2: Initialize the Project

# Initialize CDK project (TypeScript example)
cdk init app --language=typescript

# Install dependencies
npm install

Step 3: Bootstrap Your AWS Environment

Before deploying CDK applications, you need to bootstrap your AWS environment. This creates necessary resources like an S3 bucket for storing templates and IAM roles for deployments.

# Bootstrap CDK (one-time per account/region)
cdk bootstrap

>cdk bootstrap
Deploying to account: 197317184204, region: us-east-1
 ⏳  Bootstrapping environment aws://197317184204/us-east-1...
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
CDKToolkit: creating CloudFormation changeset...
CDKToolkit |  0/12 | 11:46:00 pm | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
.
.
.
CDKToolkit | 12/12 | 11:46:51 pm | CREATE_COMPLETE      | AWS::CloudFormation::Stack | CDKToolkit 
 ✅  Environment aws://197317184204/us-east-1 bootstrapped.

Step 4: Review the Infrastructure Code

Creating the VPC

The VPC is configured with 2 Availability Zones, public subnets for the load balancer, and private subnets for EC2 instances:

// Create VPC with public and private subnets
const vpc = new ec2.Vpc(this, 'WebsiteVPC', {
  maxAzs: 2,
  natGateways: 1,
  subnetConfiguration: [
    {
      name: 'PublicSubnet',
      subnetType: ec2.SubnetType.PUBLIC,
      cidrMask: 24
    },
    {
      name: 'PrivateSubnet',
      subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      cidrMask: 24
    }
  ]
});

Security Groups

We create two security groups: one for the ALB (allowing internet traffic) and one for EC2 instances (allowing traffic only from the ALB):

// Security group for ALB (public-facing)
const albSecurityGroup = new ec2.SecurityGroup(this, 'ALBSecurityGroup', {
  vpc: vpc,
  description: 'Security group for Application Load Balancer',
  allowAllOutbound: true
});

albSecurityGroup.addIngressRule(
  ec2.Peer.anyIpv4(),
  ec2.Port.tcp(80),
  'Allow HTTP traffic from internet'
);

// Security group for EC2 instances (private)
const ec2SecurityGroup = new ec2.SecurityGroup(this, 'EC2SecurityGroup', {
  vpc: vpc,
  description: 'Security group for EC2 instances',
  allowAllOutbound: true
});

ec2SecurityGroup.addIngressRule(
  albSecurityGroup,
  ec2.Port.tcp(80),
  'Allow HTTP traffic from ALB'
);

IAM Role for EC2

The IAM role allows EC2 instances to be managed via AWS Systems Manager:

// IAM role for EC2 instances
const ec2Role = new iam.Role(this, 'EC2Role', {
  assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore')
  ]
});

User Data Script

The user data script installs Apache and creates a dynamic webpage:

// User data script to install and configure web server
const userData = ec2.UserData.forLinux();
userData.addCommands(
  'yum update -y',
  'yum install -y httpd',
  'systemctl start httpd',
  'systemctl enable httpd',
  'TOKEN=$(curl --request PUT "http://169.254.169.254/latest/api/token" --header "X-aws-ec2-metadata-token-ttl-seconds: 3600")',
  'instanceId=$(curl -s http://169.254.169.254/latest/meta-data/instance-id --header "X-aws-ec2-metadata-token: $TOKEN")',
  'echo "<h1>AWS Linux VM Deployed with CDK</h1>" > /var/www/html/index.html',
  'echo "<p>Instance ID: $instanceId</p>" >> /var/www/html/index.html'
);

Creating EC2 Instances

We create 2 EC2 instances distributed across different Availability Zones:

// Create EC2 instances in different AZs
const instance1 = new ec2.Instance(this, 'WebInstance1', {
  vpc: vpc,
  instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
  machineImage: ec2.MachineImage.latestAmazonLinux2023(),
  securityGroup: ec2SecurityGroup,
  role: ec2Role,
  userData: userData,
  vpcSubnets: {
    subnets: [privateSubnets.subnets[0]]
  },
  requireImdsv2: true
});

const instance2 = new ec2.Instance(this, 'WebInstance2', {
  vpc: vpc,
  instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
  machineImage: ec2.MachineImage.latestAmazonLinux2023(),
  securityGroup: ec2SecurityGroup,
  role: ec2Role,
  userData: userData,
  vpcSubnets: {
    subnets: [privateSubnets.subnets[1]]
  },
  requireImdsv2: true
});

Application Load Balancer

Finally, we create the ALB with a target group and listener:

// Application Load Balancer
const alb = new elbv2.ApplicationLoadBalancer(this, 'WebsiteALB', {
  vpc: vpc,
  internetFacing: true,
  securityGroup: albSecurityGroup,
  vpcSubnets: {
    subnetType: ec2.SubnetType.PUBLIC
  }
});

// Target group with health checks
const targetGroup = new elbv2.ApplicationTargetGroup(this, 'WebsiteTargetGroup', {
  vpc: vpc,
  port: 80,
  protocol: elbv2.ApplicationProtocol.HTTP,
  targets: [
    new elbv2_targets.InstanceTarget(instance1, 80),
    new elbv2_targets.InstanceTarget(instance2, 80)
  ],
  healthCheck: {
    path: '/',
    interval: cdk.Duration.seconds(30)
  }
});

// Listener
alb.addListener('WebsiteListener', {
  port: 80,
  protocol: elbv2.ApplicationProtocol.HTTP,
  defaultTargetGroups: [targetGroup]
});

CloudFormation Outputs

We define outputs to easily access important resource information:

// Outputs
new cdk.CfnOutput(this, 'LoadBalancerDNS', {
  value: alb.loadBalancerDnsName,
  description: 'DNS name of the load balancer'
});

Step 5: Deploy the Infrastructure

Preview the CloudFormation template that CDK will generate:

cdk synth

Deploy the stack to AWS:

cdk deploy

CDK will show you the changes and ask for confirmation. Type y to proceed.

The deployment takes approximately 5–10 minutes. Once complete, you’ll see outputs including:

The deployment process:

CDK synthesizes your code into CloudFormation templates
Templates are uploaded to the bootstrap S3 bucket
CloudFormation creates a change set
Resources are provisioned in dependency order
Outputs are displayed (including the load balancer URL)

>cdk deploy
Deploying to account: 197317184204, region: us-east-1

✨  Synthesis time: 10.91s

WebsiteStack: start: Building WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code
WebsiteStack: success: Built WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code
WebsiteStack: start: Building WebsiteStack Template
WebsiteStack: success: Built WebsiteStack Template
WebsiteStack: start: Publishing WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code (197317184204-us-east-1-78ffe0b8)
WebsiteStack: start: Publishing WebsiteStack Template (197317184204-us-east-1-0ae3e5c9)
WebsiteStack: success: Published WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code (197317184204-us-east-1-78ffe0b8)
WebsiteStack: success: Published WebsiteStack Template (197317184204-us-east-1-0ae3e5c9)
Stack WebsiteStack
IAM Statement Changes
┌───┬──────────────────────────────────────────────────────────────┬────────┬──────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────┬───────────┐
│   │ Resource                                                     │ Effect │ Action                                                       │ Principal                                                      │ Condition │
├───┼──────────────────────────────────────────────────────────────┼────────┼──────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┼───────────┤ 
│ + │ ${Custom::VpcRestrictDefaultSGCustomResourceProvider/Role.Ar │ Allow  │ sts:AssumeRole                                               │ Service:lambda.amazonaws.com                                   │           │ 
│   │ n}                                                           │        │                                                              │                                                                │           │ 
├───┼──────────────────────────────────────────────────────────────┼────────┼──────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┼───────────┤
│ + │ ${EC2Role.Arn}                                               │ Allow  │ sts:AssumeRole                                               │ Service:ec2.amazonaws.com                                      │           │ 
├───┼──────────────────────────────────────────────────────────────┼────────┼──────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────┼───────────┤ 
│ + │ arn:aws:ec2:us-east-1:197317184204:security-group/${WebsiteV │ Allow  │ ec2:AuthorizeSecurityGroupEgress                             │ AWS:${Custom::VpcRestrictDefaultSGCustomResourceProvider/Role} │           │ 
│   │ PC.DefaultSecurityGroup}                                     │        │ ec2:AuthorizeSecurityGroupIngress                            │                                                                │           │ 
│   │                                                              │        │ ec2:RevokeSecurityGroupEgress                                │                                                                │           │ 
│   │                                                              │        │ ec2:RevokeSecurityGroupIngress                               │                                                                │           │ 
└───┴──────────────────────────────────────────────────────────────┴────────┴──────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────┴───────────┘ 
IAM Policy Changes
┌───┬────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────┐
│   │ Resource                                                   │ Managed Policy ARN                                                                           │
├───┼────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${Custom::VpcRestrictDefaultSGCustomResourceProvider/Role} │ {"Fn::Sub":"arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"} │
├───┼────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${EC2Role}                                                 │ arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore                           │
└───┴────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
Security Group Changes
┌───┬─────────────────────────────┬─────┬────────────┬─────────────────────────────┐
│   │ Group                       │ Dir │ Protocol   │ Peer                        │
├───┼─────────────────────────────┼─────┼────────────┼─────────────────────────────┤
│ + │ ${ALBSecurityGroup.GroupId} │ In  │ TCP 80     │ Everyone (IPv4)             │
│ + │ ${ALBSecurityGroup.GroupId} │ In  │ TCP 443    │ Everyone (IPv4)             │
│ + │ ${ALBSecurityGroup.GroupId} │ Out │ Everything │ Everyone (IPv4)             │
├───┼─────────────────────────────┼─────┼────────────┼─────────────────────────────┤
│ + │ ${EC2SecurityGroup.GroupId} │ In  │ TCP 80     │ ${ALBSecurityGroup.GroupId} │
│ + │ ${EC2SecurityGroup.GroupId} │ Out │ Everything │ Everyone (IPv4)             │
└───┴─────────────────────────────┴─────┴────────────┴─────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)


"--require-approval" is enabled and stack includes security-sensitive updates: 'Do you wish to deploy these changes' (y/n) y
WebsiteStack: deploying... [1/1]
WebsiteStack: creating CloudFormation changeset...
WebsiteStack |  0/39 | 11:50:52 pm | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack                | WebsiteStack User Initiated
.
.
.
WebsiteStack | 39/39 | 11:54:14 pm | CREATE_COMPLETE      | AWS::CloudFormation::Stack                | WebsiteStack 

 ✅  WebsiteStack

✨  Deployment time: 213.29s

Outputs:
WebsiteStack.Instance1Id = i-09fcc8032aee32aa0
WebsiteStack.Instance2Id = i-0c0fa16e8fbfbbbc3
WebsiteStack.LoadBalancerDNS = Websit-Websi-Ii3nPppS97V3-1650643368.us-east-1.elb.amazonaws.com
WebsiteStack.VPCId = vpc-0caf658a5e29d540d
Stack ARN:
arn:aws:cloudformation:us-east-1:197317184204:stack/WebsiteStack/8d7fe6a0-11ad-11f1-abb0-0affee6835c3

✨  Total time: 224.2s

Step 6: Test Your Application

Open the LoadBalancerDNS URL in your browser to see your website. Refresh multiple times to see traffic distributed across different instances in different availability zones.

Step 7: Explore Your Infrastructure

View the CloudFormation stack in the AWS Console:

Navigate to CloudFormation service
Find the WebsiteStack stack
Explore Resources, Events, and Outputs tabs

Cleanup

To avoid ongoing AWS charges, destroy the infrastructure when you're done:

# Delete all resources
cdk destroy

# Confirm deletion when prompted

This removes all resources created by the stack. CloudFormation ensures clean deletion in the correct order, handling dependencies automatically.

Note: The bootstrap resources (S3 bucket, IAM roles) are not deleted by cdk destroy. These are shared across all CDK applications in your account and should be kept for future use.

Project Structure

Understanding the CDK project structure helps you navigate and organize your infrastructure code effectively:

aws-cdk-website/
├── bin/
│   └── app.ts                    # CDK app entry point - defines stacks
├── lib/
│   └── website-stack.ts          # Main stack with VPC, EC2, ALB resources
├── node_modules/                 # npm dependencies (auto-generated)
├── cdk.out/                      # Synthesized CloudFormation templates
├── cdk.json                      # CDK toolkit configuration
├── cdk.context.json              # CDK context values (auto-generated)
├── package.json                  # Node.js project dependencies and scripts
├── package-lock.json             # Locked dependency versions
├── tsconfig.json                 # TypeScript compiler configuration

Key Files Explained:

bin/app.ts - Application entry point where you instantiate your CDK app and stacks:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { WebsiteStack } from '../lib/website-stack';

const app = new cdk.App();
new WebsiteStack(app, 'WebsiteStack', {
  env: { 
    account: process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEFAULT_REGION 
  }
});

lib/website-stack.ts - Contains your infrastructure definitions (VPC, EC2, ALB, security groups, etc.)

cdk.json - Configures how the CDK toolkit executes your app:

{
  "app": "npx ts-node --prefer-ts-exts bin/app.ts",
  "context": {
    "@aws-cdk/core:enableStackNameDuplicates": true,
    "aws-cdk:enableDiffNoFail": true
  }
}

package.json - Defines project metadata, dependencies, and useful scripts:

{
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "deploy": "npm run build && cdk deploy",
    "synth": "cdk synth",
    "diff": "cdk diff",
    "destroy": "cdk destroy"
  }
}

cdk.out/ - Generated directory containing CloudFormation templates after running cdk synth. This is what actually gets deployed to AWS.

Conclusion

We've successfully deployed the same highly available web infrastructure from our previous Python CDK tutorial, but this time using TypeScript. As you've seen, AWS CDK provides a consistent and powerful experience regardless of your language choice.

TypeScript brings several advantages to infrastructure as code:

Strong type safety: Catch errors at compile time before deployment
Superior IDE support: Autocomplete, inline documentation, and refactoring tools
Familiar syntax: JavaScript/TypeScript developers can leverage existing knowledge
Rich ecosystem: Access to npm packages and TypeScript tooling
Better collaboration: Type definitions serve as living documentation

Whether you choose Python, TypeScript, or another supported language, CDK transforms infrastructure provisioning from a tedious, error-prone process into an enjoyable development experience. The combination of CDK's developer-friendly interface and CloudFormation's robust deployment engine provides the best of both worlds.

References

GitHub Repository: aws-cdk-typescript
AWS CDK Documentation: https://docs.aws.amazon.com/cdk/

Building Scalable Web Application on AWS with CDK and Python

Chinmay Tonape — Tue, 24 Feb 2026 17:49:41 +0000

Introduction

Infrastructure as Code (IaC) has revolutionized how we provision and manage cloud resources. In this tutorial, we'll explore AWS Cloud Development Kit (CDK) and build a production-ready web application infrastructure with high availability built in.

What is AWS CDK?

AWS Cloud Development Kit (CDK) is an open-source software development framework that lets you define cloud infrastructure using familiar programming languages like Python, TypeScript, Java, and C#. Instead of writing verbose JSON or YAML templates, you can use the full power of programming languages to define your infrastructure.

Why Use AWS CDK?

Use familiar programming languages: Write infrastructure code in Python, TypeScript, or other supported languages
Leverage IDE features: Get autocomplete, type checking, and inline documentation
Reusable components: Create and share infrastructure patterns as libraries
Higher-level abstractions: CDK constructs provide sensible defaults and best practices
Faster development: Write less code compared to raw CloudFormation templates

CDK and CloudFormation: The Perfect Partnership

AWS CDK doesn't replace CloudFormation—it enhances it. Here's how they work together:

You write CDK code in your preferred programming language (Python in our case)
CDK synthesizes CloudFormation templates from your code using cdk synth
CloudFormation deploys the infrastructure to AWS
CloudFormation manages the stack lifecycle (updates, rollbacks, deletions)

This means you get the power of programming languages for authoring and the reliability of CloudFormation for deployment. CDK handles the complexity of generating correct CloudFormation templates, while CloudFormation ensures safe, predictable infrastructure changes.

Architecture Overview

We'll build a highly available web application with the following architecture:

┌─────────────────────────────────────────────────────────────────────┐
│                            Internet                                 │
└────────────────────────────────┬────────────────────────────────────┘
                                 │
                                 ▼
                    ┌────────────────────────┐
                    │   Internet Gateway     │
                    └────────────────────────┘
                                 │
┌────────────────────────────────┼────────────────────────────────────┐
│                                │                  VPC               │
│                                ▼                                    │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │          Application Load Balancer (ALB)                    │    │
│  │                  (Public Subnets)                           │    │
│  └──────────────────────┬──────────────────┬───────────────────┘    │
│                         │                  │                        │
│  ┌──────────────────────┼──────────────────┼─────────────────────┐  │
│  │  Availability Zone 1 │                  │ Availability Zone 2 │  │
│  │                      │                  │                     │  │
│  │  ┌───────────────────▼────────────┐  ┌──▼──────────────────┐  │  │
│  │  │   Public Subnet (AZ-1)         │  │ Public Subnet (AZ-2)│  │  │
│  │  │                                │  │                     │  │  │
│  │  │  ┌──────────────────┐          │  │                     │  │  │
│  │  │  │  NAT Gateway     │          │  │                     │  │  │
│  │  │  └────────┬─────────┘          │  │                     │  │  │
│  │  └───────────┼────────────────────┘  └─────────────────────┘  │  │
│  │              │                                                │  │
│  │  ┌───────────▼────────────────┐  ┌────────────────────────┐   │  │
│  │  │   Private Subnet (AZ-1)    │  │  Private Subnet (AZ-2) │   │  │
│  │  │                            │  │                        │   │  │
│  │  │  ┌──────────────────┐      │  │  ┌──────────────────┐  │   │  │
│  │  │  │  EC2 Instance 1  │      │  │  │  EC2 Instance 2  │  │   │  │
│  │  │  │                  │      │  │  │                  │  │   │  │
│  │  │  │                  │      │  │  │                  │  │   │  │
│  │  │  └──────────────────┘      │  │  └──────────────────┘  │   │  │
│  │  │                            │  │                        │   │  │
│  │  └────────────────────────────┘  └────────────────────────┘   │  │
│  │                                                               │  │
│  └───────────────────────────────────────────────────────────────┘  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Prerequisites

Before we begin, ensure you have:

Node.js (v14 or later) for CDK CLI
Python 3.8+ installed

Project Structure

aws-cdk-python/
│
├── app.py                    # CDK app entry point - initializes the stack
├── website_stack.py          # Main infrastructure stack definition
├── deploy.py                 # Deployment helper script
│
├── cdk.json                  # CDK configuration file
├── cdk.context.json          # CDK context values (auto-generated)
│
├── requirements.txt          # Python dependencies
├── setup.py                  # Python package setup file
│
└── cdk.out/                 # CDK synthesized CloudFormation templates (auto-generated)
    ├── WebsiteStack.template.json
    ├── manifest.json
    └── tree.json

Step 1: Install AWS CDK CLI

Install the CDK CLI globally using npm:

npm install -g aws-cdk

Verify the installation:

cdk --version

Step 2: Set Up the Project

Clone the repository or create a new directory:

git clone https://github.com/chinmayto/aws-cdk-python.git
cd aws-cdk-python

Create a Python virtual environment:

# Windows
python -m venv .venv
.venv\Scripts\activate

# Linux/Mac
python3 -m venv .venv
source .venv/bin/activate

Install Python dependencies:

pip install -r requirements.txt

Step 3: Bootstrap Your AWS Environment

CDK requires a one-time bootstrap process to create resources it needs for deployment (S3 bucket for assets, IAM roles, etc.):

cdk bootstrap

This command creates a CloudFormation stack called CDKToolkit in your AWS account.

>cdk bootstrap
Deploying to account: 197317184204, region: us-east-1
 ⏳  Bootstrapping environment aws://197317184204/us-east-1...
Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.
CDKToolkit: creating CloudFormation changeset...
CDKToolkit |  0/12 | 10:53:19 pm | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack | CDKToolkit User Initiated
.
.
.
CDKToolkit | 12/12 | 10:54:11 pm | CREATE_COMPLETE      | AWS::CloudFormation::Stack | CDKToolkit 
 ✅  Environment aws://197317184204/us-east-1 bootstrapped.

Step 4: Review the Infrastructure Code

Our infrastructure is defined in website_stack.py. Let's break down the key components:

Creating the VPC

The VPC is configured with 2 Availability Zones, public subnets for the load balancer, and private subnets for EC2 instances:

# Create VPC with public and private subnets
vpc = ec2.Vpc(
    self, "WebsiteVPC",
    max_azs=2,
    nat_gateways=1,
    subnet_configuration=[
        ec2.SubnetConfiguration(
            name="PublicSubnet",
            subnet_type=ec2.SubnetType.PUBLIC,
            cidr_mask=24
        ),
        ec2.SubnetConfiguration(
            name="PrivateSubnet",
            subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
            cidr_mask=24
        )
    ]
)

Security Groups

We create two security groups: one for the ALB (allowing internet traffic) and one for EC2 instances (allowing traffic only from the ALB):

# Security group for ALB (public-facing)
alb_security_group = ec2.SecurityGroup(
    self, "ALBSecurityGroup",
    vpc=vpc,
    description="Security group for Application Load Balancer",
    allow_all_outbound=True
)

alb_security_group.add_ingress_rule(
    ec2.Peer.any_ipv4(),
    ec2.Port.tcp(80),
    "Allow HTTP traffic from internet"
)

# Security group for EC2 instances (private)
ec2_security_group = ec2.SecurityGroup(
    self, "EC2SecurityGroup",
    vpc=vpc,
    description="Security group for EC2 instances",
    allow_all_outbound=True
)

ec2_security_group.add_ingress_rule(
    alb_security_group,
    ec2.Port.tcp(80),
    "Allow HTTP traffic from ALB"
)

IAM Role for EC2

The IAM role allows EC2 instances to be managed via AWS Systems Manager:

# IAM role for EC2 instances
ec2_role = iam.Role(
    self, "EC2Role",
    assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"),
    managed_policies=[
        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSSMManagedInstanceCore")
    ]
)

User Data Script

The user data script installs Apache and creates a dynamic webpage:

# User data script to install and configure web server
user_data = ec2.UserData.for_linux()
user_data.add_commands(
    "yum update -y",
    "yum install -y httpd",
    "systemctl start httpd",
    "systemctl enable httpd",
    "",
    "# Get instance metadata using IMDSv2",
    'TOKEN=$(curl --request PUT "http://169.254.169.254/latest/api/token" --header "X-aws-ec2-metadata-token-ttl-seconds: 3600")',
    'instanceId=$(curl -s http://169.254.169.254/latest/meta-data/instance-id --header "X-aws-ec2-metadata-token: $TOKEN")',
    'instanceAZ=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone --header "X-aws-ec2-metadata-token: $TOKEN")',
    'privHostName=$(curl -s http://169.254.169.254/latest/meta-data/local-hostname --header "X-aws-ec2-metadata-token: $TOKEN")',
    'privIPv4=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4 --header "X-aws-ec2-metadata-token: $TOKEN")',
    "",
    "# Create HTML page",
    'echo "<font face = \\"Verdana\\" size = \\"5\\">"                               > /var/www/html/index.html',
    'echo "<center><h1>AWS Linux VM Deployed with CDK using Python</h1></center>"  >> /var/www/html/index.html',
    'echo "<center> <b>EC2 Instance Metadata</b> </center>"                        >> /var/www/html/index.html',
    'echo "<center> <b>Instance ID:</b> $instanceId </center>"                     >> /var/www/html/index.html',
    'echo "<center> <b>AWS Availablity Zone:</b> $instanceAZ </center>"            >> /var/www/html/index.html',
    'echo "<center> <b>Private Hostname:</b> $privHostName </center>"              >> /var/www/html/index.html',
    'echo "<center> <b>Private IPv4:</b> $privIPv4 </center>"                      >> /var/www/html/index.html',
    'echo "</font>"                                                                >> /var/www/html/index.html'
)

Creating EC2 Instances

We create 2 EC2 instances distributed across different Availability Zones:

# Create 2 EC2 instances in private subnets across different AZs
instances = []
private_subnets = vpc.select_subnets(subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS).subnets

for i in range(2):
    instance = ec2.Instance(
        self, f"WebInstance{i+1}",
        vpc=vpc,
        instance_type=ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
        machine_image=ec2.MachineImage.latest_amazon_linux2023(),
        security_group=ec2_security_group,
        role=ec2_role,
        user_data=user_data,
        vpc_subnets=ec2.SubnetSelection(
            subnets=[private_subnets[i % len(private_subnets)]]
        ),
        require_imdsv2=True
    )
    instances.append(instance)

Application Load Balancer

Finally, we create the ALB with a target group and listener:

# Application Load Balancer
alb = elbv2.ApplicationLoadBalancer(
    self, "WebsiteALB",
    vpc=vpc,
    internet_facing=True,
    security_group=alb_security_group,
    vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PUBLIC)
)

# Target group
target_group = elbv2.ApplicationTargetGroup(
    self, "WebsiteTargetGroup",
    vpc=vpc,
    port=80,
    protocol=elbv2.ApplicationProtocol.HTTP,
    targets=[targets.InstanceIdTarget(instance.instance_id) for instance in instances],
    health_check=elbv2.HealthCheck(
        path="/",
        interval=Duration.seconds(30)
    )
)

# Listener
listener = alb.add_listener(
    "WebsiteListener",
    port=80,
    protocol=elbv2.ApplicationProtocol.HTTP,
    default_target_groups=[target_group]
)

CloudFormation Outputs

We define outputs to easily access important resource information:

# Outputs
CfnOutput(
    self, "LoadBalancerDNS",
    value=alb.load_balancer_dns_name,
    description="DNS name of the load balancer"
)

CfnOutput(
    self, "Instance1Id",
    value=instances[0].instance_id,
    description="Instance ID of the first EC2 instance"
)

CfnOutput(
    self, "Instance2Id",
    value=instances[1].instance_id,
    description="Instance ID of the second EC2 instance"
)

Step 5: Deploy the Infrastructure

Preview the CloudFormation template that CDK will generate:

cdk synth

Deploy the stack to AWS:

cdk deploy

CDK will show you the changes and ask for confirmation. Type y to proceed.

The deployment takes approximately 5-10 minutes. Once complete, you'll see outputs including:

>cdk deploy
Deploying to account: 197317184204, region: us-east-1

✨  Synthesis time: 16.13s

WebsiteStack: start: Building WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code
WebsiteStack: success: Built WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code
WebsiteStack: start: Building WebsiteStack Template
WebsiteStack: success: Built WebsiteStack Template
WebsiteStack: start: Publishing WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code (197317184204-us-east-1-78ffe0b8)
WebsiteStack: start: Publishing WebsiteStack Template (197317184204-us-east-1-2ed930b7)
WebsiteStack: success: Published WebsiteStack Template (197317184204-us-east-1-2ed930b7)
WebsiteStack: success: Published WebsiteStack/Custom::VpcRestrictDefaultSGCustomResourceProvider Code (197317184204-us-east-1-78ffe0b8)
Stack WebsiteStack
IAM Statement Changes
┌───┬─────────────────────────────────────────────────┬────────┬─────────────────────────────────────────────────┬─────────────────────────────────────────────────┬───────────┐
│   │ Resource                                        │ Effect │ Action                                          │ Principal                                       │ Condition │
├───┼─────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────┼─────────────────────────────────────────────────┼───────────┤ 
│ + │ ${Custom::VpcRestrictDefaultSGCustomResourcePro │ Allow  │ sts:AssumeRole                                  │ Service:lambda.amazonaws.com                    │           │ 
│   │ vider/Role.Arn}                                 │        │                                                 │                                                 │           │ 
├───┼─────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────┼─────────────────────────────────────────────────┼───────────┤ 
│ + │ ${EC2Role.Arn}                                  │ Allow  │ sts:AssumeRole                                  │ Service:ec2.amazonaws.com                       │           │ 
├───┼─────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────┼─────────────────────────────────────────────────┼───────────┤
│ + │ arn:aws:ec2:us-east-1:197317184204:security-gro │ Allow  │ ec2:AuthorizeSecurityGroupEgress                │ AWS:${Custom::VpcRestrictDefaultSGCustomResourc │           │ 
│   │ up/${WebsiteVPC.DefaultSecurityGroup}           │        │ ec2:AuthorizeSecurityGroupIngress               │ eProvider/Role}                                 │           │ 
│   │                                                 │        │ ec2:RevokeSecurityGroupEgress                   │                                                 │           │ 
│   │                                                 │        │ ec2:RevokeSecurityGroupIngress                  │                                                 │           │ 
└───┴─────────────────────────────────────────────────┴────────┴─────────────────────────────────────────────────┴─────────────────────────────────────────────────┴───────────┘ 
IAM Policy Changes
┌───┬────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────┐
│   │ Resource                                                   │ Managed Policy ARN                                                                           │
├───┼────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${Custom::VpcRestrictDefaultSGCustomResourceProvider/Role} │ {"Fn::Sub":"arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"} │
├───┼────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤
│ + │ ${EC2Role}                                                 │ arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore                           │
└───┴────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘
Security Group Changes
┌───┬─────────────────────────────┬─────┬────────────┬─────────────────────────────┐
│   │ Group                       │ Dir │ Protocol   │ Peer                        │
├───┼─────────────────────────────┼─────┼────────────┼─────────────────────────────┤
│ + │ ${ALBSecurityGroup.GroupId} │ In  │ TCP 80     │ Everyone (IPv4)             │
│ + │ ${ALBSecurityGroup.GroupId} │ In  │ TCP 443    │ Everyone (IPv4)             │
│ + │ ${ALBSecurityGroup.GroupId} │ Out │ Everything │ Everyone (IPv4)             │
├───┼─────────────────────────────┼─────┼────────────┼─────────────────────────────┤
│ + │ ${EC2SecurityGroup.GroupId} │ In  │ TCP 22     │ ${WebsiteVPC.CidrBlock}     │
│ + │ ${EC2SecurityGroup.GroupId} │ In  │ TCP 80     │ ${ALBSecurityGroup.GroupId} │
│ + │ ${EC2SecurityGroup.GroupId} │ Out │ Everything │ Everyone (IPv4)             │
└───┴─────────────────────────────┴─────┴────────────┴─────────────────────────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)


"--require-approval" is enabled and stack includes security-sensitive updates: 'Do you wish to deploy these changes' (y/n) y
WebsiteStack: deploying... [1/1]
WebsiteStack: creating CloudFormation changeset...
WebsiteStack |  0/39 | 10:58:15 pm | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack                | WebsiteStack User Initiated
.
.
.
WebsiteStack | 39/39 | 11:01:35 pm | CREATE_COMPLETE      | AWS::CloudFormation::Stack                | WebsiteStack

 ✅  WebsiteStack

✨  Deployment time: 213.18s

Outputs:
WebsiteStack.Instance1Id = i-089c31f66c61482a3
WebsiteStack.Instance2Id = i-0af48aefd5e4efcc2
WebsiteStack.LoadBalancerDNS = Websit-Websi-5kyWZuGmscD3-1445557942.us-east-1.elb.amazonaws.com
WebsiteStack.VPCId = vpc-002b8b79c7f0e2f49
Stack ARN:
arn:aws:cloudformation:us-east-1:197317184204:stack/WebsiteStack/33bc7c70-11a6-11f1-8888-0affc0f3836d

✨  Total time: 229.31s

Step 6: Test Your Application

Copy the LoadBalancerDNS value from the outputs and open it in your browser:

Websit-Websi-5kyWZuGmscD3-1445557942.us-east-1.elb.amazonaws.com

You should see a webpage displaying EC2 instance metadata. Refresh the page multiple times to see the load balancer distributing traffic between the two instances (notice the Instance ID and Availability Zone changing).

Step 7: Explore Your Infrastructure

View the CloudFormation stack in the AWS Console:

Navigate to CloudFormation service
Find the WebsiteStack stack
Explore Resources, Events, and Outputs tabs

Cleanup

To avoid ongoing charges, destroy the infrastructure when you're done:

cdk destroy

Type y to confirm. CDK will delete all resources created by the stack.

>cdk destroy
Deploying to account: 197317184204, region: us-east-1
Are you sure you want to delete: WebsiteStack (y/n) y
WebsiteStack: destroying... [1/1]
WebsiteStack |   0 | 11:08:57 pm | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack                | WebsiteStack User Initiated
.
.
.
WebsiteStack |  37 | 11:10:22 pm | DELETE_IN_PROGRESS   | AWS::EC2::VPC                             | WebsiteVPC (WebsiteVPCD8B49DC8)

 ✅  WebsiteStack: destroyed

Conclusion

AWS CDK transforms infrastructure provisioning from a tedious, error-prone process into an enjoyable development experience. By leveraging Python and CloudFormation together, we've built a production-ready, highly available web application infrastructure with just a few hundred lines of code.

The combination of CDK's developer-friendly abstractions and CloudFormation's robust deployment engine gives you the best of both worlds: rapid development and reliable operations.

References

GitHub Repository: aws-cdk-python
AWS CDK Documentation: https://docs.aws.amazon.com/cdk/

GitOps with ArgoCD on Amazon EKS using Terraform: A Complete Implementation Guide

Chinmay Tonape — Fri, 12 Dec 2025 04:35:31 +0000

In the rapidly evolving world of DevOps and cloud-native applications, GitOps has emerged as a revolutionary approach to continuous deployment and infrastructure management. This blog post explores how to implement a complete GitOps workflow using ArgoCD on Amazon Elastic Kubernetes Service (EKS), providing you with a production-ready setup that follows industry best practices.

GitOps represents a paradigm shift where Git repositories serve as the single source of truth for both application code and infrastructure configuration. By leveraging ArgoCD as our GitOps operator, we can achieve automated, reliable, and auditable deployments while maintaining the declarative nature of Kubernetes.

What is GitOps?

GitOps is a modern approach to continuous deployment that uses Git as the single source of truth for declarative infrastructure and applications. The core principles include:

Declarative Configuration: Everything is described declaratively in Git
Version Control: All changes are tracked and auditable
Automated Deployment: Changes in Git trigger automatic deployments
Continuous Monitoring: The system continuously ensures the desired state matches the actual state

Why ArgoCD?

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes that offers:

Application Management: Centralized management of multiple applications
Multi-Cluster Support: Deploy to multiple Kubernetes clusters
Rich UI: Intuitive web interface for monitoring deployments
RBAC Integration: Fine-grained access control
Rollback Capabilities: Easy rollback to previous versions

Architecture Overview

Our implementation creates a robust, scalable architecture that includes:

┌──────────────────────────────────────────────────────────────┐
│                        AWS Cloud                             │
│  ┌──────────────────────────────────────────────────────────┐│
│  │                    VPC                                   ││
│  │  ┌─────────────────┐    ┌──────────────────────────────┐ ││
│  │  │  Public Subnets │    │      Private Subnets         │ ││
│  │  │                 │    │  ┌─────────────────────────┐ │ ││
│  │  │  ┌───────────┐  │    │  │      EKS Cluster        │ │ ││
│  │  │  │    NAT    │  │    │  │  ┌─────────────────────┐│ │ ││
│  │  │  │  Gateway  │  │    │  │  │   NGINX Ingress     ││ │ ││
│  │  │  └───────────┘  │    │  │  │    Controller       ││ │ ││
│  │  │                 │    │  │  └─────────────────────┘│ │ ││
│  │  └─────────────────┘    │  │  ┌─────────────────────┐│ │ ││
│  │                         │  │  │     ArgoCD          ││ │ ││
│  │                         │  │  │     Server          ││ │ ││
│  │                         │  │  └─────────────────────┘│ │ ││
│  │                         │  │  ┌─────────────────────┐│ │ ││
│  │                         │  │  │   Application       ││ │ ││
│  │                         │  │  │   Workloads         ││ │ ││
│  │                         │  │  └─────────────────────┘│ │ ││
│  │                         │  └─────────────────────────┘ │ ││
│  │                         └──────────────────────────────┘ ││
│  └──────────────────────────────────────────────────────────┘│
│                                                              │
│  ┌──────────────────────────────────────────────────────────┐│
│  │                   Route53                                ││
│  │  argocd.chinmayto.com → NGINX Ingress NLB                ││
│  │  app.chinmayto.com → NGINX Ingress NLB                   ││
│  └──────────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────┘

Prerequisites

Before starting, ensure you have:

AWS CLI configured with appropriate permissions
Terraform installed (version >= 1.0)
kubectl installed
A registered domain name in Route53
Helm installed (version >= 3.0)

Implementation Steps

Step 1: Create VPC and EKS Cluster

We start by creating the foundational infrastructure using AWS community Terraform modules. First, let's define our variables:

# infrastructure/variables.tf
variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
  default     = "CT-EKS-Cluster"
}

variable "cluster_version" {
  description = "Kubernetes version for the EKS cluster"
  type        = string
  default     = "1.33"
}

variable "vpc_cidr" {
  description = "CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

variable "public_subnet_cidrs" {
  description = "CIDR blocks for public subnets"
  type        = list(string)
  default     = ["10.0.1.0/24", "10.0.2.0/24"]
}

variable "private_subnet_cidrs" {
  description = "CIDR blocks for private subnets"
  type        = list(string)
  default     = ["10.0.10.0/24", "10.0.20.0/24"]
}

Now, let's create the VPC and EKS cluster:

# infrastructure/main.tf
####################################################################################
# Data source for availability zones
####################################################################################
data "aws_availability_zones" "available" {
  state = "available"
}

####################################################################################
### VPC Module Configuration
####################################################################################
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-VPC"
  cidr = var.vpc_cidr

  azs             = slice(data.aws_availability_zones.available.names, 0, 2)
  private_subnets = var.private_subnet_cidrs
  public_subnets  = var.public_subnet_cidrs

  enable_nat_gateway = true
  enable_vpn_gateway = false
  single_nat_gateway = true

  enable_dns_hostnames = true
  enable_dns_support   = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }

  tags = {
    Name      = "${var.cluster_name}-VPC"
    Terraform = "true"
  }
}

####################################################################################
###  EKS Cluster Module Configuration
####################################################################################
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id                                   = module.vpc.vpc_id
  subnet_ids                               = module.vpc.private_subnets
  cluster_endpoint_public_access           = true
  enable_cluster_creator_admin_permissions = true

  # EKS Managed Node Groups
  eks_managed_node_groups = {
    EKS_Node_Group = {
      min_size     = 1
      max_size     = 3
      desired_size = 2

      instance_types = ["t3.medium"]
      capacity_type  = "ON_DEMAND"

      subnet_ids = module.vpc.private_subnets
    }
  }

  # EKS Add-ons
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    eks-pod-identity-agent = {
      most_recent = true
    }
  }

  tags = {
    Name      = var.cluster_name
    Terraform = "true"
  }
}

####################################################################################
###  Null Resource to update the kubeconfig file
####################################################################################
resource "null_resource" "update_kubeconfig" {
  provisioner "local-exec" {
    command = "aws eks --region ${var.aws_region} update-kubeconfig --name ${var.cluster_name}"
  }

  depends_on = [module.eks]
}

Step 2: Deploy NGINX Ingress Controller

The NGINX Ingress Controller provides external access to our services through an AWS Network Load Balancer:

# infrastructure/nginx-ingress.tf
################################################################################
# Create ingress-nginx namespace
################################################################################
resource "kubernetes_namespace" "ingress_nginx" {
  metadata {
    name = "ingress-nginx"
    labels = {
      name = "ingress-nginx"
    }
  }
  depends_on = [module.eks]
}

################################################################################
# Install NGINX Ingress Controller using Helm
################################################################################
resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = kubernetes_namespace.ingress_nginx.metadata[0].name
  version    = "4.8.3"

  values = [
    yamlencode({
      controller = {
        service = {
          type = "LoadBalancer"
          annotations = {
            "service.beta.kubernetes.io/aws-load-balancer-type"                              = "nlb"
            "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" = "true"
          }
        }
        metrics = {
          enabled = true
          serviceMonitor = {
            enabled = false
          }
        }
      }
    })
  ]

  depends_on = [kubernetes_namespace.ingress_nginx]
}

################################################################################
# Get the NLB hostname from nginx ingress controller
################################################################################
data "kubernetes_service" "nginx_ingress_controller" {
  metadata {
    name      = "ingress-nginx-controller"
    namespace = kubernetes_namespace.ingress_nginx.metadata[0].name
  }
  depends_on = [helm_release.nginx_ingress]
}

Step 3: Deploy ArgoCD Server

First, let's define the ArgoCD-specific variables:

# infrastructure/variables.tf (ArgoCD Variables)
variable "argocd_namespace" {
  description = "Kubernetes namespace for ArgoCD"
  type        = string
  default     = "argocd"
}

variable "argocd_chart_version" {
  description = "ArgoCD Helm chart version"
  type        = string
  default     = "5.51.6"
}

variable "argocd_hostname" {
  description = "Hostname for ArgoCD ingress"
  type        = string
  default     = "argocd.chinmayto.com"
}

variable "argocd_admin_password" {
  description = "Custom admin password for ArgoCD (leave empty for auto-generated)"
  type        = string
  default     = ""
  sensitive   = true
}

variable "domain_name" {
  description = "Domain name for the hosted zone"
  type        = string
  default     = "chinmayto.com"
}

variable "argocd_subdomain" {
  description = "Subdomain for ArgoCD"
  type        = string
  default     = "argocd"
}

Now, deploy ArgoCD using Helm with custom configuration for ingress access:

# infrastructure/argocd.tf
####################################################################################
### Route53 Hosted Zone
####################################################################################
data "aws_route53_zone" "main" {
  name         = var.domain_name
  private_zone = false
}

####################################################################################
### ArgoCD Namespace
####################################################################################
resource "kubernetes_namespace" "argocd" {
  metadata {
    name = var.argocd_namespace
  }

  depends_on = [module.eks]
}

####################################################################################
### ArgoCD Helm Release
####################################################################################
resource "helm_release" "argocd" {
  name       = "argocd"
  repository = "https://argoproj.github.io/argo-helm"
  chart      = "argo-cd"
  version    = var.argocd_chart_version
  namespace  = kubernetes_namespace.argocd.metadata[0].name

  values = [
    yamlencode({
      server = {
        service = {
          type = "ClusterIP"
        }
        ingress = {
          enabled          = true
          ingressClassName = "nginx"
          hosts            = [var.argocd_hostname]
          annotations = {
            "nginx.ingress.kubernetes.io/ssl-redirect"       = "false"
            "nginx.ingress.kubernetes.io/force-ssl-redirect" = "false"
            "nginx.ingress.kubernetes.io/backend-protocol"   = "HTTP"
          }
        }
      }
      configs = {
        params = {
          "server.insecure" = true
        }
      }
    })
  ]

  depends_on = [kubernetes_namespace.argocd]
}

####################################################################################
### ArgoCD Admin Password Secret (Optional - for custom password)
####################################################################################
resource "kubernetes_secret" "argocd_admin_password" {
  count = var.argocd_admin_password != "" ? 1 : 0

  metadata {
    name      = "argocd-initial-admin-secret"
    namespace = kubernetes_namespace.argocd.metadata[0].name
  }

  data = {
    password = bcrypt(var.argocd_admin_password)
  }

  depends_on = [kubernetes_namespace.argocd]
}

Step 4: Configure Route53 DNS

Set up DNS records to access ArgoCD via a custom subdomain:

# infrastructure/argocd.tf (continued)
####################################################################################
### Route53 DNS Record for ArgoCD (pointing to NGINX Ingress NLB)
####################################################################################
resource "aws_route53_record" "argocd" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = var.argocd_subdomain
  type    = "A"

  alias {
    name                   = data.kubernetes_service.nginx_ingress_controller.status.0.load_balancer.0.ingress.0.hostname
    zone_id                = "Z26RNL4JYFTOTI" # NLB zone ID for us-east-1
    evaluate_target_health = true
  }

  depends_on = [helm_release.argocd, data.kubernetes_service.nginx_ingress_controller]
}

For application DNS records, create a separate file:

# infrastructure/app-dns.tf
####################################################################################
### Route53 DNS Record for Node.js App (pointing to NGINX Ingress NLB)
####################################################################################
resource "aws_route53_record" "nodejs_app" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = var.app_subdomain
  type    = "A"

  alias {
    name                   = data.kubernetes_service.nginx_ingress_controller.status.0.load_balancer.0.ingress.0.hostname
    zone_id                = "Z26RNL4JYFTOTI" # NLB zone ID for us-east-1
    evaluate_target_health = true
  }

  depends_on = [helm_release.nginx_ingress]
}

Step 5: Create ArgoCD Project

Define an ArgoCD project to manage application deployments:

# argocd/project.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: chinmayto-apps
  namespace: argocd
spec:
  description: Project for chinmayto applications
  sourceRepos:
    - 'https://github.com/chinmayto/terraform-aws-eks-argocd.git'
  destinations:
    - namespace: '*'
      server: https://kubernetes.default.svc
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'

Step 6: Implement App-of-Apps Pattern

Create a root application that manages other applications:

# argocd/app-of-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/chinmayto/terraform-aws-eks-argocd.git
    targetRevision: HEAD
    path: argocd
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Create individual application definitions:

# argocd/nodejs-app-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nodejs-app
  namespace: argocd
spec:
  project: chinmayto-apps
  source:
    repoURL: https://github.com/chinmayto/terraform-aws-eks-argocd.git
    targetRevision: HEAD
    path: k8s-manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: simple-nodejs-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Deployment Instructions

1. Deploy Infrastructure

cd infrastructure
terraform init
terraform plan
terraform apply -auto-approve

2. Verify Deployment

# Check EKS cluster
kubectl get nodes

# Check ArgoCD pods
kubectl get pods -n argocd

# Check NGINX Ingress
kubectl get svc -n ingress-nginx

# Get ArgoCD admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

3. Access ArgoCD UI

Navigate to https://argocd.yourdomain.com and login with:

Username: admin
Password: (from step 4 above)

4. Deploy Applications

Apply the ArgoCD application manifests:

kubectl apply -f argocd/app-of-apps.yaml

Testing

Accessing argocd using the domain: http://argocd.chinmayto.com

Deployed app-of-apps:

Deployed application with components:

$ kubectl get all -n simple-nodejs-app
NAME                                        READY   STATUS    RESTARTS   AGE        
pod/deployment-nodejs-app-bfb4c4d56-pdwtt   1/1     Running   0          10m        
pod/deployment-nodejs-app-bfb4c4d56-rfxfq   1/1     Running   0          10m        

NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/service-nodejs-app   ClusterIP   172.20.120.15   <none>        80/TCP    10m

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE        
deployment.apps/deployment-nodejs-app   2/2     2            2           10m        

NAME                                              DESIRED   CURRENT   READY   AGE   
replicaset.apps/deployment-nodejs-app-bfb4c4d56   2         2         2       10m

Update the k8s-manifests, commit the code and see argocd picking up the changes and deploying them.

Deployment complete:

$ kubectl get all -n simple-nodejs-app
NAME                                         READY   STATUS    RESTARTS   AGE
pod/deployment-nodejs-app-66684f5945-fdwqp   1/1     Running   0          2m42s
pod/deployment-nodejs-app-66684f5945-h6pv9   1/1     Running   0          2m16s
pod/deployment-nodejs-app-66684f5945-tfs45   1/1     Running   0          2m30s

NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/service-nodejs-app   ClusterIP   172.20.120.15   <none>        80/TCP    13m

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/deployment-nodejs-app   3/3     3            3           13m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/deployment-nodejs-app-66684f5945   3         3         3       2m43s
replicaset.apps/deployment-nodejs-app-bfb4c4d56    0         0         0       13m

Cleanup Steps

# Verify applications present
kubectl get applications -n argocd

# Delete specific applications by name (NEVER use --all with projects)
kubectl delete application app-of-apps -n argocd
kubectl delete application nodejs-app -n argocd

# Verify applications are deleted
kubectl get applications -n argocd

# Verify custom projects
kubectl get appprojects -n argocd

# Only delete the custom project, NEVER delete default project
kubectl delete appproject chinmayto-apps -n argocd

# Verify only custom project is deleted (default should remain)
kubectl get appprojects -n argocd

# Delete the resources and namespace for application
kubectl delete all --all-namespaces -l argocd.argoproj.io/instance=nodejs-app
kubectl delete ns -l argocd.argoproj.io/instance=nodejs-app

# Destroy terraform infrastructure
cd infrastructure
terraform destroy -auto-approve

Conclusion

This implementation demonstrates a production-ready GitOps workflow using ArgoCD on Amazon EKS. By following this guide, you've created:

A secure, scalable Kubernetes cluster on AWS
Automated application deployment pipeline
Centralized application management through ArgoCD
DNS-based access to your GitOps platform

The GitOps approach with ArgoCD provides numerous benefits including improved deployment reliability, enhanced security through Git-based workflows, and simplified application lifecycle management. This foundation can be extended to support multiple environments, advanced deployment strategies, and comprehensive monitoring solutions.

References and Further Reading

GitHub Repository: terraform-aws-eks-argocd
ArgoCD Documentation: https://argo-cd.readthedocs.io/

Deploying Applications to Amazon ECS Using GitHub Actions CI/CD

Chinmay Tonape — Sun, 28 Sep 2025 08:41:09 +0000

In the earlier module, we saw how to create an Amazon ECS cluster with a task definition using AWS Community Terraform modules. Now, let’s take the next step—automating application deployment using GitHub Actions CI/CD.

By the end of this blog, you’ll understand how to set up a complete CI/CD pipeline for deploying new versions of your application to ECS, all triggered by source code changes in your repository.

What We’ll Cover

I’ve created GitHub workflows that handle the following:

initial-ecr-setup.yml – Create an Amazon ECR public repository, build the Docker image, and push it to ECR.
deploy-infrastructure.yml – Build the ECS infrastructure (cluster, services, task definitions, ALB) using Terraform.
build-and-deploy.yml – Deploy an updated version of the app whenever source code changes are pushed.
deploy-infrastructure.yml – Destroy ECS infrastructure using Terraform.

In this blog, we’ll mainly focus on the CI/CD workflow for application version upgrades.

CI/CD Pipeline for ECS Deployment

When a new version of your application is committed to the repository, the following GitHub Actions workflow ensures it is automatically built and deployed to ECS.

Secrets in GitHub Actions

For the above workflows, you’ll need to store sensitive information as GitHub Secrets. Navigate to your repository → Settings → Secrets and variables → Actions and add:

AWS_ACCESS_KEY_ID – IAM user or role access key.
AWS_SECRET_ACCESS_KEY – Corresponding secret key.

This ensures that no credentials are exposed in your workflow files.

The DevOps IAM user that will create the ECS infrastructure and run CICD pipelines will need following accesses. It will also need access to terraform state bucket and dynamodb table used for terraform state locking. Following Principle of least privilege is the need of the day!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr-public:*",
                "ecr:*",
                "ecs:*",
                "ec2:*",
                "elasticloadbalancing:*",
                "logs:*",
                "iam:*",
                "application-autoscaling:*",
                "sts:GetServiceBearerToken",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Initial Setup

To create the public ECR repository and push initial verison of the application to it use initial-ecr-setup.yml workflow and then use deploy-infrastructure.yml workflow to create ECS infrastructure with baseic version of the application.

Step 1: Build and Push Docker Image

The workflow first builds a new Docker image from your application source code and tags it with the commit SHA or a version tag. The image is then pushed to the Amazon ECR public repository.
This ensures that ECS always uses the latest available image for deployment.

name: Build and Deploy Application

on:
  push:
    paths:
      - 'app/**'
    branches:
      - main
      - develop
  pull_request:
    paths:
      - 'app/**'
    branches:
      - main

jobs:
  build-and-push:
    name: Build and Push to ECR
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.build-image.outputs.image-tag }}
      image-uri: ${{ steps.build-image.outputs.image-uri }}

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ env.AWS_REGION }}

    - name: Login to Amazon ECR Public
      id: login-ecr-public
      uses: aws-actions/amazon-ecr-login@v2
      with:
        registry-type: public
        mask-password: 'true'

    - name: Build and push Docker image
      id: build-image
      working-directory: ./app
      run: |
        # Generate image tag using git commit SHA and timestamp
        IMAGE_TAG="${GITHUB_SHA:0:8}-$(date +%s)"
        IMAGE_URI="$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"

        echo "Building image: $IMAGE_URI"

        # Build the Docker image
        docker build -t $IMAGE_URI .

        # Push the image to ECR
        docker push $IMAGE_URI

        # Also tag and push as latest
        docker tag $IMAGE_URI $ECR_REGISTRY/$ECR_REPOSITORY:latest
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:latest

Step 2: Update ECS Task Definition

ECS services run based on task definitions. To deploy a new version, we first fetch the current task definition JSON using AWS CLI. The workflow updates the container image inside the task definition JSON with the newly built image from Step 1.

    - name: Download current task definition
      run: |
        aws ecs describe-task-definition \
          --task-definition $TASK_DEFINITION \
          --query taskDefinition > task-definition.json

    - name: Update task definition with new image
      id: task-def
      uses: aws-actions/amazon-ecs-render-task-definition@v1
      with:
        task-definition: task-definition.json
        container-name: ${{ env.CONTAINER_NAME }}
        image: ${{ needs.build-and-push.outputs.image-uri }}

Step 3: Deploy to ECS Service

The new task definition is deployed to ECS using aws-actions/amazon-ecs-deploy-task-definition.

    - name: Deploy to Amazon ECS service
      uses: aws-actions/amazon-ecs-deploy-task-definition@v2
      with:
        task-definition: ${{ steps.task-def.outputs.task-definition }}
        service: ${{ env.ECS_SERVICE }}
        cluster: ${{ env.ECS_CLUSTER }}
        wait-for-service-stability: true

Accessing the Application

Once the workflow finishes, navigate to the Application Load Balancer (ALB) DNS name associated with your ECS service. You should see the new version of the application running seamlessly.

Initial ECR setup and push image:

Build ECS infrastructure:

Access initial version of application using ALB:

Now update the code, do git push and watch the build and deploy workflow run automatically!:

New image built and pushed to ECR:

ECS Service Deployemnt in progress:

Deployment complete!

Yay!!

Cleanup

When you’re done testing, you can destroy the ECS infrastructure using the Terraform cleanup workflow in destroy-infrastructure.yml

Conclusion

We successfully set up a CI/CD pipeline using GitHub Actions to deploy applications to Amazon ECS. With this automation in place:

Developers just push code, and deployment happens automatically.
ECS services always stay up-to-date with the latest version of your app.
Infrastructure and deployments are consistent, repeatable, and easy to clean up.

This approach enables faster iteration cycles and ensures your applications remain production-ready.

References

Deploying a Simple App on ECS with Fargate & Terraform using AWS Community Modules

Chinmay Tonape — Fri, 26 Sep 2025 12:10:20 +0000

Amazon Elastic Container Service (ECS) makes it easy to run containerized applications on AWS without managing servers. With AWS Fargate and Fargate Spot, you can run workloads in a serverless fashion while optimizing costs.

In this guide, we’ll use Terraform AWS Community Modules to provision the infrastructure and deploy a simple Node.js app hosted in a public Amazon ECR repository.

Architecture

At the core, we provision a VPC with both public and private subnets across multiple Availability Zones for high availability. The ECS Cluster itself is a regional construct and does not directly reside in the VPC. Instead, the actual workloads — called ECS tasks — run inside the private subnets of our VPC. Each ECS task is deployed with security groups to control inbound and outbound traffic.

To make the application accessible to users on the internet, we use an Application Load Balancer (ALB). The ALB is deployed in the public subnets, where it can receive internet traffic. It forwards requests to a target group that points to the ECS tasks running in the private subnets. This separation of public and private networking ensures security while still allowing external access through the load balancer.

The ECS task definition specifies the Node.js container image (stored in a public Amazon ECR repository), along with CPU and memory settings. It also defines port mappings so the application can listen on port 8080. The ECS service ensures the desired number of tasks are always running, and it integrates with the ALB to register and deregister tasks dynamically.

For scalability, we configure autoscaling policies based on CPU and memory utilization. This means if the load increases, ECS will automatically launch more Fargate or Fargate Spot tasks to handle the traffic, and scale down during low demand to save costs.

Step 1: Create VPC and ECS Cluster with Terraform

We’ll use the AWS community VPC and ECS modules.

################################################################################
# VPC Module
################################################################################
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.project_name}-vpc"
  cidr = var.vpc_cidr

  azs             = var.availability_zones
  private_subnets = var.private_subnets
  public_subnets  = var.public_subnets

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_vpn_gateway   = false
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = var.common_tags
}
################################################################################
# ECS Cluster Module
################################################################################
module "ecs_cluster" {
  source  = "terraform-aws-modules/ecs/aws//modules/cluster"
  version = "~> 5.0"

  cluster_name = "${var.project_name}-cluster"

  # Fargate capacity providers
  fargate_capacity_providers = {
    FARGATE = {
      default_capacity_provider_strategy = {
        weight = 50
        base   = 20
      }
    }
    FARGATE_SPOT = {
      default_capacity_provider_strategy = {
        weight = 50
      }
    }
  }

  tags = var.common_tags
}

Step 2: Create ECS Execution & Task Roles

We need IAM roles so ECS can pull images from ECR and write logs.

################################################################################
# ECS Execution Role
################################################################################
resource "aws_iam_role" "ecs_execution_role" {
  name = "${var.project_name}-ecs-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ecs-tasks.amazonaws.com"
        }
      }
    ]
  })

  tags = var.common_tags
}

resource "aws_iam_role_policy_attachment" "ecs_execution_role_policy" {
  role       = aws_iam_role.ecs_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

################################################################################
# Additional policy for ECR public registry access
################################################################################
resource "aws_iam_role_policy" "ecs_execution_ecr_policy" {
  name = "${var.project_name}-ecs-execution-ecr-policy"
  role = aws_iam_role.ecs_execution_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ecr-public:GetAuthorizationToken",
          "ecr-public:BatchCheckLayerAvailability",
          "ecr-public:GetDownloadUrlForLayer",
          "ecr-public:BatchGetImage"
        ]
        Resource = "*"
      }
    ]
  })
}

################################################################################
# ECS Task Role
################################################################################
resource "aws_iam_role" "ecs_task_role" {
  name = "${var.project_name}-ecs-task-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ecs-tasks.amazonaws.com"
        }
      }
    ]
  })

  tags = var.common_tags
}

################################################################################
# CloudWatch Logs Group
################################################################################
resource "aws_cloudwatch_log_group" "ecs_logs" {
  name              = "/ecs/${var.project_name}"
  retention_in_days = var.log_retention_days

  tags = var.common_tags
}

Step 3: Create ALB with Target Group

The ALB will live in public subnets and route traffic to ECS tasks running in private subnets.

################################################################################
# Security Group for ALB
################################################################################
resource "aws_security_group" "alb" {
  name_prefix = "${var.project_name}-alb"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = merge(var.common_tags, {
    Name = "${var.project_name}-alb-sg"
  })
}
################################################################################
# Application Load Balancer
################################################################################
resource "aws_lb" "main" {
  name               = "${var.project_name}-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = module.vpc.public_subnets

  enable_deletion_protection = false

  tags = var.common_tags
}
################################################################################
# Target Group
################################################################################
resource "aws_lb_target_group" "ecs_tg" {
  name        = "${var.project_name}-tg"
  port        = var.app_port
  protocol    = "HTTP"
  vpc_id      = module.vpc.vpc_id
  target_type = "ip"

  health_check {
    enabled             = true
    healthy_threshold   = 2
    interval            = 30
    matcher             = "200"
    path                = "/"
    port                = "traffic-port"
    protocol            = "HTTP"
    timeout             = 5
    unhealthy_threshold = 2
  }

  tags = var.common_tags
}
################################################################################
# ALB Listener
################################################################################
resource "aws_lb_listener" "web" {
  load_balancer_arn = aws_lb.main.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.ecs_tg.arn
  }

  tags = var.common_tags
}

Step 4: ECS Task Definition with Node.js Image

We’ll use a public ECR image of our Node.js app.

################################################################################
# ECS Task Definition
################################################################################
resource "aws_ecs_task_definition" "nodejs_app" {
  family                   = "${var.project_name}-task"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = tostring(var.task_cpu)
  memory                   = tostring(var.task_memory)
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn

  container_definitions = jsonencode([
    {
      name  = "nodejs-app"
      image = var.ecr_repository_url

      portMappings = [
        {
          containerPort = var.app_port
          protocol      = "tcp"
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = aws_cloudwatch_log_group.ecs_logs.name
          "awslogs-region"        = var.aws_region
          "awslogs-stream-prefix" = "ecs"
        }
      }

      environment = [
        {
          name  = "NODE_ENV"
          value = "production"
        },
        {
          name  = "PORT"
          value = tostring(var.app_port)
        }
      ]

      essential = true
    }
  ])

  tags = var.common_tags
}

Step 5: ECS Service with Autoscaling

Attach the ECS service to the ALB target group and configure autoscaling.

################################################################################
# Security Group for ECS Service
################################################################################
resource "aws_security_group" "ecs_service" {
  name_prefix = "${var.project_name}-ecs-service"
  vpc_id      = module.vpc.vpc_id

  ingress {
    from_port       = var.app_port
    to_port         = var.app_port
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = merge(var.common_tags, {
    Name = "${var.project_name}-ecs-service-sg"
  })
}
################################################################################
# ECS Service
################################################################################
module "ecs_service" {
  source  = "terraform-aws-modules/ecs/aws//modules/service"
  version = "~> 5.0"

  name        = "${var.project_name}-service"
  cluster_arn = module.ecs_cluster.arn

  # Use the separate task definition
  task_definition_arn            = aws_ecs_task_definition.nodejs_app.arn
  desired_count                  = var.min_capacity
  ignore_task_definition_changes = false
  force_new_deployment           = true
  create_task_definition         = false

  # Built-in Auto Scaling
  autoscaling_min_capacity = var.min_capacity
  autoscaling_max_capacity = var.max_capacity

  autoscaling_policies = {
    cpu = {
      policy_type = "TargetTrackingScaling"
      target_tracking_scaling_policy_configuration = {
        predefined_metric_specification = {
          predefined_metric_type = "ECSServiceAverageCPUUtilization"
        }
        target_value       = var.cpu_target_value
        scale_in_cooldown  = var.scale_in_cooldown
        scale_out_cooldown = var.scale_out_cooldown
      }
    }
    memory = {
      policy_type = "TargetTrackingScaling"
      target_tracking_scaling_policy_configuration = {
        predefined_metric_specification = {
          predefined_metric_type = "ECSServiceAverageMemoryUtilization"
        }
        target_value       = var.memory_target_value
        scale_in_cooldown  = var.scale_in_cooldown
        scale_out_cooldown = var.scale_out_cooldown
      }
    }
  }

  load_balancer = {
    service = {
      target_group_arn = aws_lb_target_group.ecs_tg.arn
      container_name   = "nodejs-app"
      container_port   = var.app_port
    }
  }

  subnet_ids = module.vpc.private_subnets
  security_group_rules = {
    alb_ingress = {
      type                     = "ingress"
      from_port                = var.app_port
      to_port                  = var.app_port
      protocol                 = "tcp"
      description              = "Service port"
      source_security_group_id = aws_security_group.alb.id
    }
    egress_all = {
      type        = "egress"
      from_port   = 0
      to_port     = 0
      protocol    = "-1"
      cidr_blocks = ["0.0.0.0/0"]
    }
  }

  service_tags = var.common_tags
}

Accessing the Application

Once deployed, Terraform will output the ALB DNS name. Open it in your browser:

Service updated to have 10 tasks:

ALB target group showing 10 IP addresse:

Cleanup

When finished, always clean up to avoid unnecessary costs:

terraform destory

Conclusion

We successfully deployed a Node.js application to ECS with Fargate and Fargate Spot using Terraform community modules.
This setup ensures:

Scalability with autoscaling policies
Cost efficiency with Fargate Spot
Security by running tasks in private subnets behind an ALB

Using Terraform makes the process repeatable, modular, and version-controlled.

References

My GitHub Repo

Terraform AWS VPC Module

Terraform AWS ECS Module

Amazon ECS Documentation

Observability on Amazon EKS Cluster: A Complete Guide to Prometheus and Grafana with Helm

Chinmay Tonape — Sun, 21 Sep 2025 18:30:08 +0000

When deploying applications on Amazon Elastic Kubernetes Service (EKS), ensuring that you can observe, monitor, and troubleshoot workloads effectively is crucial. This is where observability comes into play. In this blog, we’ll set up an EKS cluster using AWS Community Terraform modules, install Prometheus and Grafana for monitoring and visualization, configure Route53 DNS records, and deploy a sample microservices application (Voting App).## Why Prometheus and Grafana?

What is Observability in EKS?

Observability in Kubernetes (and EKS) refers to the ability to understand the internal state of your cluster and applications by analyzing the outputs such as logs, metrics, and traces. It goes beyond traditional monitoring:

Monitoring answers “Is my system working?”
Observability answers “Why is my system not working?”

In Kubernetes on AWS EKS, observability provides insights into:

Cluster health (nodes, pods, deployments, daemonsets, etc.)
Application performance (latency, error rates, resource utilization)
Infrastructure metrics (EC2, VPC, networking, EBS/EFS storage, etc.)
User experience tracking (via service-level objectives and traces)

Key pillars of observability:

Metrics – quantitative data collected over time (CPU, memory, request counts).
Logs – textual event records (pod logs, system logs).
Traces – request paths across microservices (distributed tracing).

By combining these three, DevOps teams can detect, debug, and optimize Kubernetes workloads effectively.

What is Prometheus and Grafana

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It collects time-series metrics by periodically scraping targets such as Kubernetes nodes, pods, services, and applications. Using its query language PromQL, Prometheus enables detailed analysis of cluster performance, application behavior, and infrastructure health. It also supports alerting rules, making it useful for proactive issue detection.

Grafana is an open-source visualization and analytics platform that integrates with data sources like Prometheus. It allows users to create interactive dashboards and visualizations for monitoring metrics in real time. Grafana turns raw data into meaningful insights by providing charts, graphs, and alerts that help teams detect anomalies and optimize performance.

Together, Prometheus and Grafana form a powerful observability stack on EKS. Prometheus handles metrics collection and storage, while Grafana provides visualization and analysis. This combination allows Kubernetes operators and developers to not only monitor cluster health but also understand why issues occur, enabling faster debugging and better capacity planning.

Architecture Overview

Our monitoring architecture consists of several key components:

┌─────────────────────────────────────────────────────────────┐
│                        EKS Cluster                          │
│                                                             │
│  ┌─────────────────┐    ┌─────────────────┐                 │
│  │   Prometheus    │◄───┤  Node Exporter  │                 │
│  │     Server      │    │                 │                 │
│  │                 │    └─────────────────┘                 │
│  │  - Scrapes      │                                        │
│  │  - Stores       │    ┌─────────────────┐                 │
│  │  - Alerts       │◄───┤ kube-state-     │                 │
│  └─────────────────┘    │ metrics         │                 │
│           │             └─────────────────┘                 │
│           │                                                 │
│           ▼              ┌─────────────────┐                │
│  ┌─────────────────┐◄────┤   Application   │                │
│  │    Grafana      │     │    Metrics      │                │
│  │                 │     └─────────────────┘                │
│  │  - Dashboards   │                                        │
│  │  - Visualization│     ┌─────────────────┐                │
│  │  - Alerting     │◄────┤  AlertManager   │                │
│  └─────────────────┘     │                 │                │
│                          └─────────────────┘                │
└─────────────────────────────────────────────────────────────┘

Architecture of Prometheus and Grafana on EKS

When deploying Prometheus and Grafana on EKS, the architecture typically looks like this:

Prometheus runs inside the cluster and is responsible for:
- Scraping metrics from Kubernetes components, node exporters, pods, and application endpoints.
- Storing time-series data in its own storage backend.
- Exposing APIs that allow querying using PromQL.
Grafana runs alongside Prometheus and is responsible for:
- Connecting to Prometheus as a data source.
- Querying metrics with PromQL queries.
- Creating dashboards that visualize cluster health, workload performance, and application-specific data.

Data Flow:

Kubernetes Metrics Sources:

Kubelet – node and pod metrics.
Kube-state-metrics – state of Kubernetes objects (deployments, pods, nodes, services).
cAdvisor – container-level CPU, memory, filesystem, and network usage.
ETCD metrics – key-value store performance.
Application endpoints – any service exposing /metrics in Prometheus format.

Prometheus Scraping:

Prometheus periodically queries these endpoints using service discovery (Kubernetes API, annotations, labels).
Metrics are collected and stored as time-series data.

Grafana Visualization:

Grafana queries Prometheus using PromQL.
Dashboards are built to visualize CPU usage, memory utilization, pod restarts, latency, and custom application metrics.
Dashboards can be customized or imported from Grafana’s community templates.

Step 1: Create EKS Cluster using AWS Community Terraform Modules

The first step in integrating Amazon EFS with EKS is to provision a Kubernetes cluster that runs securely inside a dedicated VPC. We will use the widely adopted AWS Terraform community modules for both the VPC and EKS setup. Please refer to main module of GitHub repo.

Step 2: Add Helm Repositories

First, let's add the necessary Helm repositories:

# Add Prometheus community Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# Add Grafana Helm repository  
helm repo add grafana https://grafana.github.io/helm-charts

# Add Nginx Ingress Helm repository  
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

# Update repositories
helm repo update

Step 3: Install NGINX Ingress Controller with Helm

To expose Prometheus, Grafana, and other applications securely outside the cluster, we need an Ingress Controller. We’ll use the NGINX Ingress Controller, installed via Helm in its own namespace.

################################################################################
# Create ingress-nginx namespace
################################################################################
resource "kubernetes_namespace" "ingress_nginx" {
  metadata {
    name = "ingress-nginx"
    labels = {
      name = "ingress-nginx"
    }
  }
  depends_on = [module.eks]
}

################################################################################
# Install NGINX Ingress Controller using Helm
################################################################################
resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  namespace  = kubernetes_namespace.ingress_nginx.metadata[0].name
  version    = "4.8.3"

  values = [
    yamlencode({
      controller = {
        service = {
          type = "LoadBalancer"
          annotations = {
            "service.beta.kubernetes.io/aws-load-balancer-type"                              = "nlb"
            "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" = "true"
          }
        }
        metrics = {
          enabled = true
          serviceMonitor = {
            enabled = false
          }
        }
      }
    })
  ]

  depends_on = [kubernetes_namespace.ingress_nginx]
}

################################################################################
# Get the NLB hostname from nginx ingress controller
################################################################################
data "kubernetes_service" "nginx_ingress_controller" {
  metadata {
    name      = "ingress-nginx-controller"
    namespace = kubernetes_namespace.ingress_nginx.metadata[0].name
  }
  depends_on = [helm_release.nginx_ingress]
}

Step 4: Create Route53 Records for Prometheus and Grafana

To make these tools accessible via a domain, configure Route53 DNS records. This allows you to access Prometheus at https://prometheus.chinmayto.com and Grafana at https://grafana.chinmayto.com.

################################################################################
# Get Route53 hosted zone for chinmayto.com
################################################################################
data "aws_route53_zone" "main" {
  name         = "chinmayto.com"
  private_zone = false
}

################################################################################
# Create Route53 A records for Prometheus and Grafana
################################################################################
resource "aws_route53_record" "prometheus" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = "prometheus.chinmayto.com"
  type    = "A"

  alias {
    name                   = data.kubernetes_service.nginx_ingress_controller.status.0.load_balancer.0.ingress.0.hostname
    zone_id                = "Z26RNL4JYFTOTI" # NLB zone ID for us-east-1
    evaluate_target_health = true
  }

  depends_on = [helm_release.nginx_ingress]
}

resource "aws_route53_record" "grafana" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = "grafana.chinmayto.com"
  type    = "A"

  alias {
    name                   = data.kubernetes_service.nginx_ingress_controller.status.0.load_balancer.0.ingress.0.hostname
    zone_id                = "Z26RNL4JYFTOTI" # NLB zone ID for us-east-1
    evaluate_target_health = true
  }

  depends_on = [helm_release.nginx_ingress]
}

Step 5: Install Prometheus Stack

We’ll use the Terraform Helm provider to install both Prometheus (for metrics collection) and Grafana (for visualization) in its own namespace.

The kube-prometheus-stack Helm chart is an umbrella chart maintained by the Prometheus community. It includes Prometheus, Grafana, and kube-state-metrics all bundled together. That’s why deploying just this chart is usually sufficient for Kubernetes observability.

################################################################################
# Create monitoring namespace
################################################################################
resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
    labels = {
      name = "monitoring"
    }
  }
  depends_on = [module.eks]
}

################################################################################
# Install Prometheus using Helm (after nginx ingress)
################################################################################
resource "helm_release" "prometheus" {
  name       = "prometheus"
  repository = "https://prometheus-community.github.io/helm-charts"
  chart      = "kube-prometheus-stack"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  version    = "55.5.0"

  values = [
    yamlencode({
      prometheus = {
        prometheusSpec = {
          retention = "30d"
        }
        service = {
          type = "ClusterIP"
        }
        ingress = {
          enabled          = true
          ingressClassName = "nginx"
          hosts            = ["prometheus.chinmayto.com"]
          paths            = ["/"]
          annotations = {
            "nginx.ingress.kubernetes.io/rewrite-target" = "/"
          }
        }
      }
      grafana = {
        enabled       = true
        adminPassword = "admin123"
        service = {
          type = "ClusterIP"
        }
        ingress = {
          enabled          = true
          ingressClassName = "nginx"
          hosts            = ["grafana.chinmayto.com"]
          path             = "/"
          annotations = {
            "nginx.ingress.kubernetes.io/rewrite-target" = "/"
          }
        }
        persistence = {
          enabled = false
        }
      }
      alertmanager = {
        enabled = true
      }
    })
  ]

  depends_on = [kubernetes_namespace.monitoring, helm_release.nginx_ingress]
}

Step 6: Deploy Sample Microservices Application (Voting App)

Now that observability is in place, let’s deploy a sample microservices app: the Voting App (a simple app with a frontend, backend, and database).

Apply manifests:

cd example-voting-app/k8s-specifications
kubectl apply -f .

Accessing Your Monitoring Stack

You can access Prometheus at https://prometheus.chinmayto.com and Grafana at https://grafana.chinmayto.com.

Validation Steps

1. Verify Prometheus Targets

In Prometheus UI:

Go to Status → Targets
Ensure all targets are "UP"
Check for any scraping errors

2. Test Prometheus Queries

Try these sample queries in Prometheus:

# CPU usage by node
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Pod count by namespace
count by (namespace) (kube_pod_info)

3. Verify Grafana Dashboards

Grafana comes with pre-built dashboards:

Navigate to Dashboards
Check "Kubernetes / Compute Resources / Cluster"
Verify data is displaying correctly

Few Best Grafana Dashboards for EKS Monitoring

Grafana provides a wide range of prebuilt dashboards for Kubernetes and Prometheus. Some of the most popular ones for EKS monitoring include:

Kubernetes / Compute Resources Cluster (ID: 315): Visualizes cluster-wide CPU and memory usage.
Kubernetes / Compute Resources Namespace (Workloads) (ID: 3146): Breaks down metrics by namespace, useful for multi-team environments.
Kubernetes / Compute Resources Pod (ID: 7633): Pod-level monitoring (CPU, memory, restarts).
Kubernetes API Server Metrics (ID: 12006): Helps track API server performance, latency, and error rates.
Node Exporter Full (ID: 1860): Deep-dive into node-level metrics such as disk I/O, CPU, memory, and network.
ETCD Metrics (ID: 3070): Monitoring the control plane’s key-value store.
Kube-State-Metrics Dashboards (ID: 13332): Tracks states of Kubernetes objects like deployments, daemonsets, jobs, and cronjobs. These dashboards can be directly imported into Grafana using their Dashboard IDs from Grafana Labs Dashboard Repository

Cleanup

Delete the k8s resources created

cd example-voting-app/k8s-specifications
kubectl delete -f .

And then terraform destroy the EKS infrastructure if you are not using it to save costs.

Conclusion

Setting up Prometheus and Grafana on EKS provides powerful observability into your Kubernetes clusters. This monitoring stack enables you to:

Track cluster and application performance
Set up proactive alerting
Visualize metrics through beautiful dashboards
Make data-driven decisions about scaling and optimization

The combination of Prometheus's robust metric collection and Grafana's visualization capabilities creates a comprehensive monitoring solution that scales with your infrastructure needs.

Remember to regularly review and update your monitoring configuration as your applications and infrastructure evolve. Monitoring is not a "set it and forget it" solution – it requires ongoing attention and refinement.

References

Kubernetes Storage Playlist - Part 5: Choosing the Right Storage for EKS: EBS vs. EFS vs. S3

Chinmay Tonape — Thu, 11 Sep 2025 02:30:01 +0000

When running applications on Amazon EKS (Elastic Kubernetes Service), one of the first architectural questions you’ll face is: Which storage solution should I use?

Picking the right storage isn’t just about saving data—it directly impacts scalability, reliability, and cost-efficiency of your workloads. AWS provides multiple storage services that integrate with Kubernetes through CSI drivers, and each serves a different purpose.

In this post, we’ll compare the three primary options — Amazon EBS, Amazon EFS, and Amazon S3 (via the Mountpoint CSI driver) — to help you choose the right fit for your workloads.

Amazon EBS (Elastic Block Store)

Amazon EBS is block-level storage attached to a specific EC2 instance or pod through the EBS CSI driver. Think of it like attaching a hard drive to your VM — it’s fast, consistent, and designed for single-pod workloads.

Best suited for:

Relational & NoSQL Databases (PostgreSQL, MySQL, Cassandra)
Stateful applications needing low-latency, high IOPS
Single-instance apps requiring persistence across pod restarts

Key Characteristics:

AZ-bound: An EBS volume lives in one Availability Zone, and pods must be scheduled in the same AZ.
Single-pod access: Most volumes are ReadWriteOnce (RWO) — meaning one pod at a time.
Dynamic provisioning: Kubernetes can auto-create EBS volumes using PersistentVolumeClaims.

If your workload is a database or something that needs dedicated, fast disk storage, EBS is the go-to.

Amazon EFS (Elastic File System)

Amazon EFS is a fully managed, elastic NFS storage that multiple pods can mount simultaneously — even across different Availability Zones. With the EFS CSI driver, it becomes a shared, scalable storage layer for Kubernetes workloads.

Best suited for:

Web content management (WordPress, Drupal)
CI/CD pipelines (shared build artifacts, code repos)
Applications needing shared data across multiple pods
Persistent storage with ReadWriteMany (RWX) access

Key Characteristics:

Shared access: Thousands of pods can read/write to the same file system.
Elastic & serverless: Automatically grows/shrinks with usage.
Multi-AZ durability: Data is stored across multiple AZs for resilience.

If you need shared storage accessible by many pods at once, EFS is the right choice.

Amazon S3 with Mountpoint CSI Driver

Amazon S3 is object storage, but with the Mountpoint for S3 CSI driver, you can mount an S3 bucket into your pod as if it were a file system. This is great for workloads that want S3’s scale and durability but expect a file system interface.

Best suited for:

Data analytics & ML jobs reading/writing large datasets
Centralized log aggregation from multiple pods
Backups & archival storage
Media processing and high-throughput workloads

Key Characteristics:

Object store with file-like access: File operations are translated into S3 API calls.
High throughput: Great for parallel data processing.
Static provisioning only: You can’t dynamically create S3 buckets — only mount existing ones.
Not POSIX-compliant: Doesn’t support all file operations like EBS/EFS.

If your workload is data-intensive and benefits from S3’s scale and economics, the Mountpoint driver is an excellent fit.

Feature	Amazon EBS	Amazon EFS	Amazon S3
Type	Block storage	File system (NFS)	Object storage (via CSI)
Access Mode	Single pod (RWO)	Multi-pod (RWX)	Multi-pod (limited)
Performance	Low latency, high IOPS	Shared, elastic, multi-AZ	High throughput, sequential
Use Case	Databases, stateful apps	Shared web content, CI/CD, dev tools	Data analytics, logs, backups
Provisioning	Static & dynamic	Static & dynamic	Static only
AZ Scope	Single AZ	Multi-AZ	Regional (bucket-level)

Conclusion

Choosing the right storage solution in EKS depends on your workload needs:

Use EBS for high-performance, single-pod workloads like databases.
Use EFS for shared, scalable storage where multiple pods need simultaneous access.
Use S3 (Mountpoint) when you need massive, cost-effective object storage for data-heavy applications.

By aligning your storage choice with your workload requirements, you’ll build an EKS architecture that’s resilient, cost-efficient, and scalable.

References

Refer to my detailed blogs for implementing each of the storage type

AWS Storage with EFS

AWS Storage with EBS

AWS Storage with S3 with Mountpoint

Building an Amazon EKS Cluster with raw Terraform Resources

Chinmay Tonape — Wed, 10 Sep 2025 18:55:42 +0000

Most guides for provisioning Amazon EKS use the AWS community Terraform module for convenience. While this is great for speed, it often abstracts away what’s actually happening behind the scenes.

In this blog, we will build an EKS cluster from scratch using raw Terraform resources (without using the community module). By doing this, you’ll gain a clear understanding of the AWS networking, IAM, and compute resources that make up a Kubernetes cluster on AWS.

Finally, we’ll deploy a sample microservices application (Voting App) on the cluster and access it using port-forwarding.

Architecture

Here’s how the setup works at a high level:

VPC is created with 2 Availability Zones for high availability.
Each AZ contains both a public and a private subnet.
EKS worker nodes (EC2 instances) are launched in private subnets for better security.
A NAT Gateway is provisioned in a public subnet to allow worker nodes in private subnets to pull images and updates from the internet (e.g., from ECR, Docker Hub).
EKS control plane (managed by AWS) communicates with the worker nodes securely within the VPC. This setup ensures that your nodes are not directly exposed to the internet while still having outbound internet access via the NAT gateway.

Step 1: Create a VPC with Public and Private Subnets

EKS requires a properly configured VPC with both public and private subnets. Worker nodes generally live in private subnets, while the NAT gateway in public subnets provides outbound internet connectivity.

We also add Internet Gateway (IGW), NAT Gateway, and route tables to handle connectivity between public/private subnets.

################################################################################
# VPC
################################################################################
resource "aws_vpc" "custom_vpc" {
  cidr_block = var.networking.cidr_block

  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-VPC"
  })
}

################################################################################
# PUBLIC SUBNETS
################################################################################
resource "aws_subnet" "public_subnets" {
  count                   = var.networking.public_subnets == null || var.networking.public_subnets == "" ? 0 : length(var.networking.public_subnets)
  vpc_id                  = aws_vpc.custom_vpc.id
  cidr_block              = var.networking.public_subnets[count.index]
  availability_zone       = var.networking.azs[count.index]
  map_public_ip_on_launch = true

  tags = merge(var.common_tags, {
    Name                                        = "${var.naming_prefix}-public-subnet-${count.index}"
    "kubernetes.io/role/elb"                    = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  })
}

################################################################################
# PRIVATE SUBNETS
################################################################################
resource "aws_subnet" "private_subnets" {
  count                   = var.networking.private_subnets == null || var.networking.private_subnets == "" ? 0 : length(var.networking.private_subnets)
  vpc_id                  = aws_vpc.custom_vpc.id
  cidr_block              = var.networking.private_subnets[count.index]
  availability_zone       = var.networking.azs[count.index]
  map_public_ip_on_launch = false

  tags = merge(var.common_tags, {
    Name                                        = "${var.naming_prefix}-private-subnet-${count.index}"
    "kubernetes.io/role/internal-elb"           = "1"
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  })
}

################################################################################
# INTERNET GATEWAY
################################################################################
resource "aws_internet_gateway" "i_gateway" {
  vpc_id = aws_vpc.custom_vpc.id

  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-i-gateway"
  })
}

################################################################################
# EIPs
################################################################################
resource "aws_eip" "elastic_ip" {
  count      = var.networking.private_subnets == null || var.networking.nat_gateways == false ? 0 : length(var.networking.private_subnets)
  depends_on = [aws_internet_gateway.i_gateway]

  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-eip-${count.index}"
  })
}

################################################################################
# NAT GATEWAYS
################################################################################
resource "aws_nat_gateway" "nats" {
  count             = var.networking.private_subnets == null || var.networking.nat_gateways == false ? 0 : length(var.networking.private_subnets)
  subnet_id         = aws_subnet.public_subnets[count.index].id
  connectivity_type = "public"
  allocation_id     = aws_eip.elastic_ip[count.index].id
  depends_on        = [aws_internet_gateway.i_gateway]

  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-nat-gw-${count.index}"
  })
}

################################################################################
# PUBLIC ROUTE TABLE
################################################################################
resource "aws_route_table" "public_table" {
  vpc_id = aws_vpc.custom_vpc.id
}

resource "aws_route" "public_routes" {
  route_table_id         = aws_route_table.public_table.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.i_gateway.id
}

resource "aws_route_table_association" "assoc_public_routes" {
  count          = length(var.networking.public_subnets)
  subnet_id      = aws_subnet.public_subnets[count.index].id
  route_table_id = aws_route_table.public_table.id
}

################################################################################
# PRIVATE ROUTE TABLES
################################################################################
resource "aws_route_table" "private_tables" {
  count  = length(var.networking.azs)
  vpc_id = aws_vpc.custom_vpc.id
}

resource "aws_route" "private_routes" {
  count                  = length(var.networking.private_subnets)
  route_table_id         = aws_route_table.private_tables[count.index].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.nats[count.index].id
}

resource "aws_route_table_association" "assoc_private_routes" {
  count          = length(var.networking.private_subnets)
  subnet_id      = aws_subnet.private_subnets[count.index].id
  route_table_id = aws_route_table.private_tables[count.index].id
}

################################################################################
# SECURITY GROUPS
################################################################################
resource "aws_security_group" "sec_groups" {
  for_each    = { for sec in var.security_groups : sec.name => sec }
  name        = "${var.naming_prefix}-${each.value.name}"
  description = each.value.description
  vpc_id      = aws_vpc.custom_vpc.id

  dynamic "ingress" {
    for_each = try(each.value.ingress, [])
    content {
      description      = ingress.value.description
      from_port        = ingress.value.from_port
      to_port          = ingress.value.to_port
      protocol         = ingress.value.protocol
      cidr_blocks      = ingress.value.cidr_blocks
      ipv6_cidr_blocks = ingress.value.ipv6_cidr_blocks
    }
  }

  dynamic "egress" {
    for_each = try(each.value.egress, [])
    content {
      description      = egress.value.description
      from_port        = egress.value.from_port
      to_port          = egress.value.to_port
      protocol         = egress.value.protocol
      cidr_blocks      = egress.value.cidr_blocks
      ipv6_cidr_blocks = egress.value.ipv6_cidr_blocks
    }
  }
}

Step 2: Create IAM Roles and Policies

EKS needs roles for both the control plane and worker nodes.

Cluster Role – Grants permissions for EKS to manage AWS resources.
Node Role – Allows EC2 instances (worker nodes) to join the cluster.

################################################################################
# EKS CLUSTER ROLE
################################################################################
resource "aws_iam_role" "EKSClusterRole" {
  name = "${var.naming_prefix}-EKSClusterRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "eks.amazonaws.com"
        }
      },
    ]
  })
}

// This policy provides Kubernetes the permissions it requires to manage resources on your behalf. 
// Kubernetes requires Ec2:CreateTags permissions to place identifying information on EC2 resources
// including but not limited to Instances, Security Groups, and Elastic Network Interfaces
resource "aws_iam_role_policy_attachment" "AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.EKSClusterRole.name
}

################################################################################
# NODE GROUP ROLE
################################################################################
resource "aws_iam_role" "NodeGroupRole" {
  name = "${var.naming_prefix}-EKSNodeGroupRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })
}

//This policy allows Amazon EKS worker nodes to connect to Amazon EKS Clusters.
resource "aws_iam_role_policy_attachment" "AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.NodeGroupRole.name
}

// Provides read-only access to Amazon EC2 Container Registry repositories.
resource "aws_iam_role_policy_attachment" "AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.NodeGroupRole.name
}

// This policy provides the Amazon VPC CNI Plugin (amazon-vpc-cni-k8s) the permissions
// it requires to modify the IP address configuration on your EKS worker nodes. 
// This permission set allows the CNI to list, describe, and modify ENIs on your behalf
resource "aws_iam_role_policy_attachment" "AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.NodeGroupRole.name
}

Step 3: Provision the EKS Cluster

Following creates the EKS control plane.

################################################################################
# EKS Cluster
################################################################################
resource "aws_eks_cluster" "eks-cluster" {
  name     = var.cluster_config.name
  role_arn = aws_iam_role.EKSClusterRole.arn
  version  = var.cluster_config.version

  vpc_config {
    subnet_ids         = flatten([var.public_subnets_id, var.private_subnets_id])
    security_group_ids = flatten(var.security_groups_id)
  }

  depends_on = [
    aws_iam_role_policy_attachment.AmazonEKSClusterPolicy
  ]

}

At this point, we create a managed set of EC2 worker nodes with basic required addons

################################################################################
# NODE GROUP
################################################################################
resource "aws_eks_node_group" "node-ec2" {
  for_each        = { for node_group in var.node_groups : node_group.name => node_group }
  cluster_name    = aws_eks_cluster.eks-cluster.name
  node_group_name = "${var.naming_prefix}-${each.value.name}"
  node_role_arn   = aws_iam_role.NodeGroupRole.arn
  subnet_ids      = flatten(var.private_subnets_id)

  scaling_config {
    desired_size = try(each.value.scaling_config.desired_size, 2)
    max_size     = try(each.value.scaling_config.max_size, 3)
    min_size     = try(each.value.scaling_config.min_size, 1)
  }

  update_config {
    max_unavailable = try(each.value.update_config.max_unavailable, 1)
  }

  ami_type       = each.value.ami_type
  instance_types = each.value.instance_types
  capacity_type  = each.value.capacity_type
  disk_size      = each.value.disk_size
  version        = var.cluster_config.version

  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-nodes-${each.value.name}"
  })

  depends_on = [
    aws_iam_role_policy_attachment.AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.AmazonEC2ContainerRegistryReadOnly,
    aws_iam_role_policy_attachment.AmazonEKS_CNI_Policy
  ]
}

resource "aws_iam_openid_connect_provider" "eks_oidc_provider" {
  url             = aws_eks_cluster.eks-cluster.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["9e99a48a9960b14926bb7f3b02e22da2b0ab7280"]
}


################################################################################
# EKS Addons
################################################################################
data "aws_eks_addon_version" "eks_addons" {
  for_each = var.addons

  addon_name         = each.value.name
  kubernetes_version = aws_eks_cluster.eks-cluster.version
  most_recent        = coalesce(each.value.most_recent, true)
}

resource "aws_eks_addon" "addons" {
  for_each = var.addons

  cluster_name                = aws_eks_cluster.eks-cluster.name
  addon_name                  = each.value.name
  addon_version               = coalesce(each.value.version, data.aws_eks_addon_version.eks_addons[each.key].version)
  resolve_conflicts_on_create = "OVERWRITE"

  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-addon-${each.value.name}"
  })

  depends_on = [aws_eks_node_group.node-ec2]
}

Step 4: Configure kubectl and Access Cluster

################################################################################
# Update kubeconfig with created cluster
################################################################################
resource "null_resource" "update_kubeconfig" {
  provisioner "local-exec" {
    command = "aws eks --region ${var.aws_region} update-kubeconfig --name ${var.cluster_config.name}"
  }
  depends_on = [module.eks]
}

Step 5: Deploy a Sample Microservices Application

We will deploy famous exmaple voting app from the https://github.com/dockersamples/example-voting-app. We will just deploy the manifests and not look into building the docker images at this moment.

A front-end web app in Python which lets you vote between two options
A Redis which collects new votes
A .NET worker which consumes votes and stores them in…
A Postgres database backed by a Docker volume
A Node.js web app which shows the results of the voting in real time

Note: The voting application only accepts one vote per client browser. It does not register additional votes if a vote has already been submitted from a client.

Apply the manifests and verify

cd /example-voting-app/k8s-specifications
kubectl apply -f .
deployment.apps/db created
service/db created
deployment.apps/redis created
service/redis created
deployment.apps/result created
service/result created
deployment.apps/vote created
service/vote created
deployment.apps/worker created


kubectl port-forward service/vote 8080:8080
kubectl port-forward service/result 8081:8081

Access app using:

Vote:   http://localhost:8080
Result: http://localhost:8081

Cleanup

Delete the k8s resources created

cd /example-voting-app/k8s-specifications
kubectl delete -f .

And then terraform destroy the EKS infrastructure if you are not using it to save costs.

Conclusion

By building an EKS cluster from scratch with Terraform—without relying on the AWS community module—we gain a clear understanding of the essential AWS resources required to run Kubernetes in production.

We saw how networking forms the foundation, how the cluster and node groups tie everything together, and finally how workloads are deployed and accessed. This exercise provides not only a strong foundation in AWS and Kubernetes but also the confidence to customize infrastructure for real-world production use cases.

References

Kubernetes Storage Playlist - Part 4: Implementing Amazon S3 Storage with EKS using Terraform and and Kubernetes Manifests

Chinmay Tonape — Mon, 08 Sep 2025 19:07:09 +0000

In this blog, we’ll explore how to integrate Amazon S3 as a storage solution with Amazon EKS using Terraform and Kubernetes YAML manifests. We will run a simple Nginx container that serves website files stored in an S3 bucket.

This approach leverages the Mountpoint for S3 CSI driver, which provides Kubernetes workloads access to Amazon S3 objects using standard POSIX interfaces.

Understanding Amazon S3 for Kubernetes

What is Amazon S3?

Amazon Simple Storage Service (Amazon S3) is an object storage service designed for scalability, durability, and availability. Unlike traditional block storage (like EBS) or file storage (like EFS), S3 stores data as objects inside buckets, which makes it ideal for static content, logs, and backups.

Architecture Diagram

The architecture of attaching Amazon S3 storage to an EKS cluster revolves around the S3 CSI (Container Storage Interface) driver and the Mountpoint for S3 integration.

At the base layer, an S3 bucket acts as the backend object store where application data is kept. Inside Kubernetes, a StorageClass defines how storage is provisioned and consumed. With S3, however, only static provisioning is currently supported—meaning a PersistentVolume (PV) must be manually created and mapped to an existing S3 bucket, and then a PersistentVolumeClaim (PVC) binds to that PV so workloads can access it.

Once a pod is deployed, Kubernetes mounts the PVC to the container’s filesystem through the S3 CSI driver, which internally uses Mountpoint for S3 to provide file system-like access to objects in the bucket.

While dynamic provisioning—where PVCs automatically trigger creation of new storage volumes—is common for drivers like EBS and EFS, it is not yet available for S3. This makes static provisioning the only option, requiring administrators to pre-define the mapping between PVs and S3 buckets. Security is handled through IAM Roles for Service Accounts (IRSA), ensuring pods have only the minimum required permissions to access specific S3 buckets.

Key Benefits of Using S3 with EKS

Scalability: Virtually unlimited storage capacity.
Durability: 11 9’s durability guarantees.
Cost-Effective: Pay for what you use with no upfront provisioning.
Integration: Easy integration with AWS services like Athena, Glue, and CloudFront.

Important Considerations for EKS

CSI Driver Requirements: The Mountpoint for S3 CSI driver must be installed.
Pod Identity Support: Currently, IRSA (IAM Roles for Service Accounts) is required. Pod Identity is not supported yet.
Limitations: Only static provisioning is supported as of now. Dynamic provisioning is on the roadmap.
Resource Quotas: Watch for limits like open file descriptors and network throughput when scaling workloads.

Step 1: Provisioning EKS Cluster with Terraform in a VPC

The first step in integrating Amazon S3 with EKS is to provision a Kubernetes cluster that runs securely inside a dedicated VPC. We will use the widely adopted AWS Terraform community modules for both the VPC and EKS setup. Please refer to main module of GitHub repo.

Step 2: Creating the S3 bucket

S3 Bucket: Encrypted S3 bucket with versioning and permissions

####################################################################################
# S3 Bucket for Kubernetes Storage
####################################################################################
resource "aws_s3_bucket" "main" {
  bucket = var.bucket_name

  tags = {
    Name        = var.bucket_name
    Environment = var.environment
    Terraform   = "true"
  }
}

####################################################################################
# S3 Bucket Versioning
####################################################################################
resource "aws_s3_bucket_versioning" "main" {
  bucket = aws_s3_bucket.main.id
  versioning_configuration {
    status = var.versioning_enabled ? "Enabled" : "Disabled"
  }
}

####################################################################################
# S3 Bucket Encryption
####################################################################################
resource "aws_s3_bucket_server_side_encryption_configuration" "main" {
  bucket = aws_s3_bucket.main.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
    bucket_key_enabled = true
  }
}

####################################################################################
# S3 Bucket Public Access Block
####################################################################################
resource "aws_s3_bucket_public_access_block" "main" {
  bucket = aws_s3_bucket.main.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

IAM Role: For Mountpoint for S3 CSI Driver

####################################################################################
# IAM Role for S3 CSI Driver (IRSA)
####################################################################################
resource "aws_iam_role" "s3_csi_driver_role" {
  name = "${var.cluster_name}-s3-csi-driver-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:sub" = "system:serviceaccount:kube-system:s3-csi-driver-sa"
            "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:aud" = "sts.amazonaws.com"
          }
        }
      }
    ]
  })

  tags = {
    Name        = "${var.cluster_name}-s3-csi-driver-role"
    Environment = var.environment
    Terraform   = "true"
  }
}

####################################################################################
# S3 CSI Driver Policy (Based on AWS Documentation)
####################################################################################
resource "aws_iam_policy" "s3_csi_driver_policy" {
  name        = "${var.cluster_name}-s3-csi-driver-policy"
  description = "IAM policy for S3 CSI Driver based on AWS documentation"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = aws_s3_bucket.main.arn
      },
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:AbortMultipartUpload",
          "s3:DeleteObject"
        ]
        Resource = "${aws_s3_bucket.main.arn}/*"
      }
    ]
  })

  tags = {
    Name        = "${var.cluster_name}-s3-csi-driver-policy"
    Environment = var.environment
    Terraform   = "true"
  }
}

####################################################################################
# Attach S3 CSI Driver Policy
####################################################################################
resource "aws_iam_role_policy_attachment" "s3_csi_driver_policy" {
  role       = aws_iam_role.s3_csi_driver_role.name
  policy_arn = aws_iam_policy.s3_csi_driver_policy.arn
}

Mountpoint for S3 Add-on: AWS Mountpoint for S3 CSI Driver for Kubernetes integration

####################################################################################
###  S3 Mountpoint CSI Driver Addon (deployed after S3 module)
####################################################################################
resource "aws_eks_addon" "s3_mountpoint_csi_driver" {
  cluster_name             = module.eks.cluster_name
  addon_name               = "aws-mountpoint-s3-csi-driver"
  service_account_role_arn = module.s3.s3_csi_driver_role_arn
  # Using IRSA for CSI driver, Pod Identity for application pods

  # Ensure this addon is created after the S3 module creates the IAM role and pod identity association
  depends_on = [module.s3]

  tags = {
    Name        = "${var.cluster_name}-s3-mountpoint-csi-driver"
    Environment = var.environment
    Terraform   = "true"
  }
}

Step 3: S3 Storage Implementation Patterns

Since only static provisioning is supported for S3 today, we’ll define PersistentVolumes (PV) that reference an S3 bucket.

storage-class.yaml - StorageClass for S3 CSI driver (requires ${S3_BUCKET_NAME} placeholder)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: s3-csi-sc
provisioner: s3.csi.aws.com
parameters:
  bucketName: ${S3_BUCKET_NAME}
  prefix: "k8s-storage/"
volumeBindingMode: Immediate
allowVolumeExpansion: false

persistent-volume.yaml - PersistentVolume for S3 bucket (static provisioning only)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: s3-csi-sc
  csi:
    driver: s3.csi.aws.com
    volumeHandle: s3-csi-driver-volume
    volumeAttributes:
      bucketName: ${S3_BUCKET_NAME}
      prefix: "k8s-storage/"

persistent-volume-claim.yaml - PVC using S3 CSI driver

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: s3-csi-sc
  resources:
    requests:
      storage: 1Gi

nginx-pod.yaml - Nginx pod with S3 storage mounted and content creation

apiVersion: v1
kind: Pod
metadata:
  name: nginx-s3-pod
  namespace: default
  labels:
    app: nginx-s3
spec:
  securityContext:
    runAsUser: 0
    runAsGroup: 0
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    command: ["/bin/sh", "-c"]
    args:
    - |
      echo '<h1>Hello from S3 Mountpoint!</h1><p><b>Pod:</b> '$POD_NAME'</p>' > /usr/share/nginx/html/index.html || true
      sed -i 's/user  nginx;/user  root;/' /etc/nginx/nginx.conf
      nginx -g 'daemon off;'
    volumeMounts:
    - name: s3-storage
      mountPath: /usr/share/nginx/html
  volumes:
  - name: s3-storage
    persistentVolumeClaim:
      claimName: s3-pvc

nginx-service.yaml - ClusterIP service to expose nginx on port 80

apiVersion: v1
kind: Service
metadata:
  name: nginx-s3-service
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx-s3

Deployment Steps For Static Provisioning:

Get S3 bucket name from Terraform:

cd infrastructure
S3_BUCKET_NAME=$(terraform output -raw s3_bucket_name 2>/dev/null || echo "")

Update manifests with S3 values:

sed "s/\${S3_BUCKET_NAME}/$S3_BUCKET_NAME/g" storage-class.yaml > storage-class-final.yaml

sed "s/\${S3_BUCKET_NAME}/$S3_BUCKET_NAME/g" persistent-volume.yaml > persistent-volume-final.yaml

Apply manifests:

kubectl apply -f storage-class-final.yaml
kubectl apply -f persistent-volume-final.yaml
kubectl apply -f persistent-volume-claim.yaml
kubectl apply -f nginx-pod.yaml
kubectl apply -f nginx-service.yaml

Refer to deploy.sh for automated deployment script

$ kubectl get sc,pv,pvc
NAME                                    PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/s3-csi-sc   s3.csi.aws.com          Delete          Immediate              false                  10m

NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/s3-pv   1Gi        RWX            Retain           Bound    default/s3-pvc   s3-csi-sc      <unset>                          10m

NAME                           STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/s3-pvc   Bound    s3-pv    1Gi        RWX            s3-csi-sc      <unset>                 10m

S3 Bucket:

S3 Bucket Contents:

Nginx pod accessing S3 for index.html

Verification

# Check S3 CSI driver status:
kubectl get pods -n kube-system -l app=aws-mountpoint-s3-csi-driver

# Check deployment status:
kubectl get pod nginx-s3-pod
kubectl get service nginx-s3-service
kubectl get pvc s3-pvc
kubectl get pv

# Test nginx web server:
kubectl port-forward service/nginx-s3-service 8084:80
# Then visit http://localhost:8084

# Check S3 File Persistence
kubectl exec nginx-s3-pod -- ls -la /usr/share/nginx/html/
kubectl exec nginx-s3-pod -- cat /usr/share/nginx/html/index.html

# Check S3 bucket content:
aws s3 ls s3://$S3_BUCKET_NAME/k8s-storage/

Cleanup

kubectl delete -f nginx-service.yaml
kubectl delete -f nginx-pod.yaml
kubectl delete -f persistent-volume-claim.yaml
kubectl delete -f persistent-volume.yaml
kubectl delete -f storage-class.yaml

And then terraform destroy the EKS infrastructure if you are not using it to save costs.

Conclusion

Amazon S3 provides a scalable and cost-effective storage solution for workloads running on EKS. With the Mountpoint for S3 CSI driver, Kubernetes pods can directly mount S3 buckets and serve static files seamlessly. While currently limited to static provisioning, this integration is a powerful way to manage object storage in containerized environments.

In this blog, we provisioned an EKS cluster using Terraform, set up IAM roles with IRSA, deployed the S3 CSI driver, and ran an Nginx container backed by S3 storage.