Production-Ready AWS EKS with Terraform, GitOps, DevSecOps & FinOps
Spinning up a textbook production Kubernetes cluster on AWS is deceptively expensive.
Between the Amazon EKS control plane ($73.00/month), redundant multi-AZ NAT Gateways ($65.70/month), On-Demand EC2 worker nodes ($60.74/month), and cross-AZ data transfer fees, an idle baseline cluster can easily burn $240.00/month before serving a single real user.
Worse, once a team starts shipping Terraform changes through Pull Requests, a single unreviewed instance resize or additional NAT Gateway can increase the monthly AWS bill overnight.
In Part 1 of the CloudPulse series, I built a hardened, non-root FastAPI chaos microservice and a shift-left DevSecOps pipeline that eliminated 3 HIGH-severity container CVEs before publishing the verified image to GHCR.
In Part 2, I built the production AWS landing zone, FinOps cost-governance gate, Argo CD GitOps controller, and Prometheus/Grafana observability stack to run that container in production.
The result? A 38.1% reduction in estimated baseline AWS infrastructure spending, bringing monthly costs down from $240.00 to $148.60.
This article breaks down the complete four-layer AWS architecture and how every component is enforced automatically through CI/CD.
Part 2 Repository: cloudpulse-eks-gitops-infra
Architecture Overview: The Four Production Layers
Why separate the application repository (cloudpulse-devsecops-service) from the infrastructure repository (cloudpulse-eks-gitops-infra)?
Following the enterprise GitOps pattern prevents infinite CI/CD loops, isolates IAM permissions between application developers and platform engineers, and creates a clean audit trail where every infrastructure change is a standalone Git commit.
Our infrastructure repository wraps the verified container image in four layers:
Layer 1 — AWS Cloud Networking, EKS Compute & IAM (Terraform)
A custom 10.0.0.0/16 VPC across ap-south-1a and ap-south-1b, a KMS-encrypted Amazon EKS v1.31 cluster, EC2 Spot/hybrid managed node groups, and scoped IAM roles.
Layer 2 — Shift-Left FinOps & IaC Security Gates (GitHub Actions)
Automated terraform validate, Trivy IaC scanning, Kubeconform strict schema checks, and a custom Python FinOps script that blocks Pull Requests exceeding a $250.00/month AWS budget.
Layer 3 — Zero-Touch GitOps & Self-Healing (Argo CD + Kubernetes HPA)
Continuous reconciliation watching k8s/base/ with selfHeal: true and prune: true, paired with a HorizontalPodAutoscaler scaling from 2 to 6 replicas during traffic surges.
Layer 4 — Real-Time Flash-Sale Observability (Prometheus + Grafana)
Custom PromQL telemetry tracking request rates, pod readiness, and flash-sale surge events with a 5-second dashboard refresh interval.
Layer 1: AWS VPC Networking, EKS Compute & IAM
Building the AWS VPC with Terraform
In terraform/vpc.tf, we provision an isolated 10.0.0.0/16 Virtual Private Cloud in ap-south-1, spanning two Availability Zones.
The network consists of:
-
2 Public Subnets (
10.0.1.0/24and10.0.2.0/24): Tagged withkubernetes.io/role/elbfor public ingress load balancers and routed through an Internet Gateway. -
2 Private Subnets (
10.0.10.0/24and10.0.20.0/24): Tagged withkubernetes.io/role/internal-elb, where EKS worker nodes run without direct public internet access. - Zero-Trust Security Groups & VPC Flow Logs: The default VPC security group is stripped of all rules, control-plane ingress is restricted to HTTPS within the VPC CIDR, and network traffic is logged to Amazon CloudWatch Logs with KMS encryption and 30-day retention.
AWS VPC Resource Map
This is the AWS Console VPC Resource Map provisioned through Terraform, showing the VPC, public and private subnets across two Availability Zones, route tables, and Internet Gateway routing.
FinOps Networking Optimization: Shared Single-AZ NAT Gateway
A standard multi-AZ NAT Gateway deployment costs $65.70/month in fixed hourly charges, in addition to cross-AZ data processing fees.
For this workload tier, I implemented a Single-AZ Shared NAT Gateway + Elastic IP architecture, routing egress traffic (container image pulls from ghcr.io and AWS API calls) from both private subnets through a single optimized NAT tier.
This architectural choice:
- Cuts fixed NAT costs by 50% (-$32.85/month).
- Reduces cross-AZ data transfer overhead by 86.5% (-$22.11/month).
Hardened Amazon EKS v1.31 & Least-Privilege IAM
Rather than relying on overly permissive default IAM policies, terraform/iam.tf provisions three strictly scoped IAM roles:
1. control-plane-role (eks.amazonaws.com)
Attached exclusively to AmazonEKSClusterPolicy and AmazonEKSVPCResourceController.
2. worker-node-role (ec2.amazonaws.com)
Scoped strictly to worker node container runtime and CNI networking policies.
3. vpc-flow-log-role (vpc-flow-logs.amazonaws.com)
Scoped only to publishing network audit streams to CloudWatch.
AWS IAM Roles Overview
Three isolated IAM roles created through Terraform, separating control-plane, worker-node, and VPC Flow Logs permissions.
In terraform/eks.tf, the Amazon EKS v1.31 cluster (cloudpulse-eks-prod) and its managed node group enforce defense-in-depth at the infrastructure layer.
KMS Envelope Encryption
Kubernetes secrets are encrypted at rest using a dedicated AWS KMS Customer Managed Key with automatic yearly key rotation enabled.
Private-Only API Endpoint & Control Plane Logging
All five EKS control-plane audit and diagnostic log streams are enabled by default.
Hardened EC2 Launch Templates
Worker nodes enforce IMDSv2 (http_tokens = "required" to block SSRF metadata attacks), run on encrypted gp3 EBS volumes with no public IPs, and utilize EC2 Spot (t3.medium) capacity to cut compute spending by 60% (-$36.44/month).
AWS IAM Control Plane Role Verification
This screenshot verifies the control-plane role and its two attached AWS-managed policies: AmazonEKSClusterPolicy and AmazonEKSVPCResourceController.
Layer 2: Shift-Left FinOps & IaC Security Gates
Most teams only discover cloud cost drift when the AWS invoice arrives at the end of the month.
To shift FinOps left, I wrote a zero-dependency Python 3 cost-governance scanner (scripts/finops_cost_guard.py) that runs on every Pull Request inside .github/workflows/iac-gitops-ci.yml.
Alongside Terraform validation (terraform fmt and validate), Trivy IaC security scanning (blocking HIGH or CRITICAL misconfigurations), and Kubeconform Kubernetes v1.31 schema validation, the FinOps guard parses terraform/*.tf and calculates the monthly run rate before a single resource can be merged.
Baseline vs. Optimized AWS Cost Breakdown
| AWS Resource | Architecture Configuration | Baseline Cost | Optimized Cost | FinOps Savings |
|---|---|---|---|---|
| Amazon EKS Control Plane | EKS v1.31 (cloudpulse-eks-prod) |
$73.00 | $73.00 | $0.00 |
| AWS NAT Gateway | Single-AZ Shared NAT vs. Multi-AZ | $65.70 | $32.85 | -$32.85 (-50.0%) |
| EC2 Worker Nodes | 2x t3.medium (Spot / Hybrid) |
$60.74 | $24.30 | -$36.44 (-60.0%) |
| Amazon EBS Storage | 100 GB Encrypted gp3 (2x 50GB) |
$8.00 | $8.00 | $0.00 |
| CloudWatch Logs | VPC Flow Logs + EKS Audit (30d) | $6.00 | $6.00 | $0.00 |
| AWS KMS Key | Secrets Envelope Encryption | $1.00 | $1.00 | $0.00 |
| Network Data Transfer | Cross-AZ Optimization + Shared EIP | $25.56 | $3.45 | -$22.11 (-86.5%) |
| TOTAL MONTHLY SPEND | AWS ap-south-1 |
$240.00 | $148.60 | -$91.40 (-38.1%) |
If a developer opens a Pull Request that pushes estimated monthly spending past our $250.00/month policy ceiling, finops_cost_guard.py exits with code 1 and blocks the merge.
At $148.60/month, we maintain $101.40/month (40.6%) of budget headroom to absorb HPA autoscaling bursts during peak traffic.
Automated CI/CD Governance
The GitHub Actions pipeline executes four quality and governance checks:
-
Terraform Format & Validation: Runs
terraform fmt -check, initializes Terraform without a backend, and executesterraform validate. - FinOps Cost Governance: Scans Terraform configurations and enforces the $250.00/month budget policy.
- Trivy IaC Security: Scans infrastructure configurations for HIGH and CRITICAL security misconfigurations.
- Kubernetes Manifest Validation: Uses Kubeconform strict schema validation and Kustomize checks against Kubernetes v1.31.
Any failing gate blocks the Pull Request from being merged.
Layer 3: Argo CD GitOps Delivery, Self-Healing & HPA Autoscaling
Once infrastructure passes all four CI gates, Argo CD (argocd/application.yaml) takes over continuous delivery.
Nobody runs manual kubectl apply commands against production.
The GitOps controller continuously watches the infrastructure repository and reconciles Kubernetes resources with the declared Git state.
The Argo CD configuration:
syncPolicy:
automated:
prune: true # Automatically deletes resources removed from Git
selfHeal: true # Reverts manual kubectl edits back to Git state
syncOptions:
- CreateNamespace=true
Argo CD Application Tree & Sync Status
The Argo CD Application Details Tree for cloudpulse-production shows the deployed Kubernetes resources, including the Namespace, Service, Deployment, ReplicaSet, Pods, and HPA.
The application is synced to the main branch, with automated synchronization, pruning, and self-healing enabled.
Self-Healing & Dynamic Flash-Sale Scaling
Inside k8s/base/deployment.yaml, our pods run with strict Kubernetes security contexts:
runAsNonRoot: truerunAsUser: 10001readOnlyRootFilesystem: true-
drop: ["ALL"]capabilities seccompProfile: RuntimeDefault
Liveness and readiness probes hit GET /healthz on port 8000 every 10 seconds.
If a container returns HTTP 503 for three consecutive checks during an outage simulation (/simulate-outage), Kubernetes replaces the unhealthy pod while healthy replicas continue serving traffic.
Meanwhile, our HorizontalPodAutoscaler (hpa.yaml) monitors CPU utilization against a 60% target threshold, scaling smoothly between:
minReplicas: 2maxReplicas: 6
Kubernetes Cluster & HPA Verification
This terminal snapshot demonstrates the Kubernetes workload validation, including node readiness, running application pods, replica availability, and HPA configuration.
The local K3s validation environment is also used to reproduce the Kubernetes deployment topology without continuously running paid AWS infrastructure.
Layer 4: Real-Time Flash-Sale Observability
Visualizing Traffic Surges with Prometheus & Grafana
Every pod includes Prometheus scrape annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: "/metrics"
When we trigger a flash-sale traffic spike, Prometheus scrapes custom counters and Grafana (monitoring/grafana-cloudpulse-dashboard.json) visualizes the surge through a 5-second rolling dashboard.
Grafana Production Telemetry Dashboard
The dashboard tracks:
- 32.6K total HTTP requests served.
- 51 flash-sale surge events triggered.
- HEALTHY (UP) service availability.
- 2 active Kubernetes pods ready.
The Active Flash-Sale Traffic Rate graph shows traffic ramping up to approximately 300 requests per second, with two distinct traffic peaks.
The /api/v1/checkout endpoint peaks at 194 requests per second, while /healthz probe traffic reaches 1.35 requests per second.
The dashboard uses PromQL queries to track request rates and surge events in real time.
Try It Yourself: Full Source Code
To avoid burning money on idle AWS control planes outside live benchmarks, you can run the entire Terraform, Trivy, Kubeconform, and FinOps validation pipeline locally.
You can also spin up the full topology on a local k3d cluster with Argo CD and Grafana at zero AWS infrastructure cost.
Part 2 — AWS EKS, Terraform, FinOps Gate & Argo CD
Part 1 — FastAPI Microservice & Hardened Docker CI/CD
Both repositories contain the source code and configuration needed to explore the complete CloudPulse platform.
Built by Mohammed Armaan Kazi
Cloud Infrastructure • GitOps • Platform Engineering • DevSecOps • FinOps
Top comments (0)