Building a Production-Grade Microservices Platform on AWS EKS

🎯 Introduction

Running microservices in development is easy. Building a production-grade platform with hard limits, zero-trust pod IAM roles, automated GitOps delivery, full telemetry, and an SRE incident response framework is where the real engineering lives.

This guide walks through the end-to-end deployment of a 3-tier polyglot microservice stack (Go, Node.js, Python) sourced from the OpenTelemetry Astronomy Shop demo onto AWS EKS (v1.31).

What We're Building:

1. Hardened Containers: Multi-stage distroless builds (up to 97% size reduction).
2. Infrastructure as Code: Terraform modules for VPC, EKS, and IRSA (IAM Roles for Service Accounts).
3. Declarative GitOps: ArgoCD auto-sync paired with GitHub Actions CI.
4. Production Observability: Prometheus, Grafana, custom SLI rules, and alerting thresholds.
5. Incident Response: A simulated Sev-1 Redis OOMKill outage with a complete post-mortem.
6. Cost Engineering: Destroy-on-demand strategy using AWS EC2 SPOT instances.

---

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                         INTERNET                                │
└────────────────────────┬────────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────────┐
│                       AWS eu-west-1                             │
├─────────────────────────────────────────────────────────────────┤
│  ┌───────────────────────────────────────────────────────────┐ │
│  │               VPC (10.0.0.0/16)                           │ │
│  │  ┌────────────────────────┐   ┌────────────────────────┐ │ │
│  │  │ Public Subnets (2 AZs) │   │ Private Subnets(2 AZs) │ │ │
│  │  └────────────────────────┘   └────────────────────────┘ │ │
│  │                                                           │ │
│  │  ┌─────────────────────────────────────────────────────┐ │ │
│  │  │  EKS Cluster (Kubernetes 1.31)                      │ │ │
│  │  │  Node Group: 2x m7i-flex.large SPOT                 │ │ │
│  │  │                                                     │ │ │
│  │  │  ┌──────────────────────────────────────────────┐   │ │ │
│  │  │  │ namespace: ecommerce                        │   │ │ │
│  │  │  │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │   │ │ │
│  │  │  │ │ Checkout   │ │ Payment    │ │ Recommend  │ │   │ │ │
│  │  │  │ │ (Go:5050)  │ │(Node:50051)│ │ (Py:9001)  │ │   │ │ │
│  │  │  │ └────────────┘ └────────────┘ └────────────┘ │   │ │ │
│  │  │  └──────────────────────────────────────────────┘   │ │ │
│  │  │                                                     │ │ │
│  │  │  ┌──────────────────────┐ ┌──────────────────────┐  │ │ │
│  │  │  │ ArgoCD (GitOps)      │ │ Monitoring           │  │ │ │
│  │  │  │ (Auto-sync)          │ │ (Prometheus/Grafana) │  │ │ │
│  │  │  └──────────────────────┘ └──────────────────────┘  │ │ │
│  │  └─────────────────────────────────────────────────────┘ │ │
│  └───────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

---

🐳 Step 1: Multi-Stage Distroless Docker Builds

Single-stage Docker images contain build tools, compilers, and shell utilities that bloat storage and widen your attack surface.

By switching to Multi-Stage Builds and Google Distroless Runtimes, we strip everything except the compiled binary and essential runtime dependencies.

Example: Checkout Service (Go)

# --- Stage 1: Build ---
FROM golang:1.22-alpine AS builder
WORKDIR /usr/src/app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/checkout .

# --- Stage 2: Runtime ---
FROM gcr.io/distroless/static-debian12
WORKDIR /usr/src/app
COPY --from=builder /out/checkout ./checkout
USER 65532
EXPOSE 5050
ENTRYPOINT ["./checkout"]

Optimization Results

Service	Language	Initial Size	Optimized Size	Size Reduction
Checkout	Go	1.34 GB	37.2 MB	97.2% ⬇️
Payment	Node.js	1.71 GB	265 MB	84.5% ⬇️
Recommendation	Python	276 MB	238 MB	13.8% ⬇️

Pod Security Hardening

Every Kubernetes deployment enforces non-root execution and immutable root filesystems:

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

---

🏗️ Step 2: Infrastructure as Code (Terraform)

The infrastructure uses modularized Terraform state stored remotely with S3 encryption and DynamoDB state locking.

# main.tf
module "vpc" {
  source               = "./modules/vpc"
  vpc_cidr             = "10.0.0.0/16"
  public_subnet_cidrs  = ["10.0.1.0/24", "10.0.2.0/24"]
  private_subnet_cidrs = ["10.0.10.0/24", "10.0.11.0/24"]
  cluster_name         = "ecommerce-eks"
}

module "eks" {
  source          = "./modules/eks"
  cluster_name    = "ecommerce-eks"
  cluster_version = "1.31"

  node_groups = {
    general = {
      instance_types = ["m7i-flex.large"]
      capacity_type  = "SPOT" # ~70% cost reduction
      scaling_config = {
        desired_size = 2
        max_size     = 3
        min_size     = 1
      }
    }
  }
}

Zero-Trust Access via IRSA (IAM Roles for Service Accounts)

Instead of attaching permanent IAM policies to worker nodes, pods assume targeted IAM roles via OIDC tokens:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/[oidc.eks.eu-west-1.amazonaws.com/id/](https://oidc.eks.eu-west-1.amazonaws.com/id/)<OIDC_ID>"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "[oidc.eks.eu-west-1.amazonaws.com/id/](https://oidc.eks.eu-west-1.amazonaws.com/id/)<OIDC_ID>:sub": "system:serviceaccount:kube-system:aws-load-balancer-controller"
      }
    }
  }]
}

---

☸️ Step 3: Kubernetes Deployments & Service Discovery

Microservices communicate internally using CoreDNS naming conventions (<service-name>.<namespace>.svc.cluster.local).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: ecommerce
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      serviceAccountName: checkout
      containers:
      - name: checkout
        image: neshandoc/checkout:latest
        ports:
        - containerPort: 5050
        env:
        - name: PAYMENT_ADDR
          value: "payment:50051"
        - name: PRODUCT_CATALOG_ADDR
          value: "recommendation:9001"
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          tcpSocket:
            port: 5050
          initialDelaySeconds: 10
        readinessProbe:
          tcpSocket:
            port: 5050
          initialDelaySeconds: 5

---

🔄 Step 4: CI/CD Pipeline & GitOps Flow

We enforce a strict separation between code delivery (GitHub Actions) and cluster state reconciliation (ArgoCD).

[ Developer Push ] ──► [ GitHub Actions ] ──► Builds & Pushes Image to Registry
                                                    │
                                                    ▼
[ EKS Cluster ] ◄── [ ArgoCD Auto-Sync ] ◄── Updates Kubernetes Manifest Tag

ArgoCD Application Spec

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ecommerce
  namespace: argocd
spec:
  project: default
  source:
    repoURL: [https://github.com/Neshanpro/ecommerce-microservices-app](https://github.com/Neshanpro/ecommerce-microservices-app)
    targetRevision: main
    path: kubernetes
  destination:
    server: [https://kubernetes.default.svc](https://kubernetes.default.svc)
    namespace: ecommerce
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

---

📊 Step 5: Observability, SLIs, and Alerting

We deployed the kube-prometheus-stack via Helm to track cluster metrics, pod resource limits, and application-level Service Level Indicators (SLIs).

Prometheus Alert Rules (alerts.yaml)

groups:
  - name: service_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: (sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100 > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"

      - alert: PodCrashLooping
        expr: kube_pod_container_status_waiting{reason="CrashLoopBackOff"} > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Pod {{ $labels.pod }} is in CrashLoopBackOff"

---

🚨 Step 6: Simulated Sev-1 Incident Response

To test our observability stack, we intentionally misconfigured the Redis cache memory limit (100Mi) to trigger an Out-Of-Memory termination during load testing.

Impact Matrix

Metric	Normal State	During Incident	Post-Resolution
Checkout Pod Status	2/2 Running	0/2 CrashLoopBackOff	2/2 Running
Transaction Success	99.9%	~50.0%	99.9%
P95 Latency	150ms	2000ms+	150ms

Post-Mortem Timeline

14:30:00 UTC - Redis cache memory usage exceeds 100Mi; OOMKilled by Linux Kernel.
14:30:30 UTC - Checkout pods enter CrashLoopBackOff due to dropped connection.
14:31:00 UTC - Alert `PodCrashLooping` triggers in Prometheus. On-call acknowledges.
14:33:00 UTC - Engineer runs `kubectl describe pod redis-xxx` -> Exit Code 137 confirmed.
14:34:00 UTC - Hotfix applied: Redis memory limit updated from 100Mi to 256Mi.
14:35:00 UTC - Redis recovers; Checkout service auto-reconnects. All probes passing.

---

💰 Step 7: Cost Optimization Engineering

Running a 24/7 EKS cluster for personal projects burns credits quickly. I implemented a Destroy-On-Demand Lifecycle:

Deploy (20m) ──► Test & Capture Telemetry (30m) ──► Tear Down (15m)

• Cost per test session: ~$0.85 (utilizing EC2 SPOT instances for worker nodes).
• Estimated 24/7 cost: ~$116.00/month reduced to pennies per execution block.

---

🛠️ Lessons Learned & Best Practices

1. Distroless Requires Exact Execution Paths: Distroless images lack shell utilities (/bin/sh). Script entrypoints or uncompiled shebang paths must be explicitly resolved at build time.
2. Graceful Failures over Tight Coupling: The Checkout outage proved that microservices must implement retry backoffs and circuit breakers so cache failure doesn't crash downstream dependencies.
3. Stateless Operations with GitOps: Storing application state declaratively in Git means recovering from a total EKS cluster crash takes less than 15 minutes.

---

📂 Source Code & Resources

• 🔗 GitHub Repository: Neshanpro/ecommerce-microservices-app
• 📖 OpenTelemetry Demo: opentelemetry-demo
• 🛠️ Helm Charts: kube-prometheus-stack