DEV Community

Cover image for Lessons from My DevOps Capstone: Troubleshooting AWS EKS, Free Tier Limits, and Observability
John
John

Posted on

Lessons from My DevOps Capstone: Troubleshooting AWS EKS, Free Tier Limits, and Observability

#kubernetes #terraform #aws #devops

DevOps Capstone Projects: Cohort Field Guide and EKS Troubleshooting Notes

Hey everyone,

I've seen many of us in the group hitting the same roadblocks with AWS EKS, weird Kubernetes pod scheduling errors, memory crashes, and the fact that several starter configs were not provided by the instructor.

I spent time debugging these exact problems hands-on on AWS while completing both Capstone 1 (the Voting App) and Capstone 2 (Stan's Robot Shop) from scratch. To help everyone get through smoothly without getting stuck on the same blockers, I documented every single issue I ran into, why it happened, and the exact commands and configurations that fixed it.

I also organized all the starter configs into a clean starter kit repo so you have working templates to reference and use directly.

Starter Kit & Configuration Files:

All the ready-to-use Terraform modules, Helm charts, Dockerfiles, and observability manifests mentioned in this guide are open-sourced in the companion repository:

devops-capstone-starter-kit on GitHub

If this guide helps you through the capstone, please drop a star on the repo and follow me on GitHub for more cloud and DevOps guides!

1. What Was Missing (Starter Configs Created From Scratch)

The assignment prompt mentions starter configurations, but since they were not provided, we had to build them from the ground up. Here is the full list of what had to be created from scratch:

For Capstone 1 (Voting App):

  1. Terraform EKS Infrastructure: Complete Terraform module to provision an AWS EKS v1.30 cluster (versions.tf, variables.tf, main.tf, outputs.tf) on AWS Free Tier eligible instances (t3.small) inside the default VPC.
  2. Container Hardening: Added non-root user execution (appuser, UID 10001) in worker/Dockerfile and pinned modern, secure base images across all services (python:3.11-slim, node:18-slim, dotnet:7.0).
  3. CI/CD Pipeline with Trivy Scanning: GitHub Actions workflow (.github/workflows/deploy.yml) that builds images, runs Aqua Security Trivy for CVE scans, and publishes immutable image tags (${{ github.sha }}) to Docker Hub.
  4. Helm Chart: Complete Helm package (voting-app/) with Chart.yaml, values.yaml, and Kubernetes deployment/service manifests for all 5 services (vote, result, redis, db, worker), plus rollback testing.

For Capstone 2 (Robot Shop):

  1. Clean Microservices Helm Deployment: Stripped all proprietary Instana agent dependencies from the services (web, catalogue, cart, payment, user), converted Redis storage to emptyDir so it runs without AWS EBS CSI drivers, and added the Locust load-gen deployment into Helm for continuous traffic.
  2. Prometheus & Grafana RED Metrics Dashboard: Provisioned Prometheus and Grafana using kube-prometheus-stack with a custom RED metrics dashboard (Rate, Errors, Duration) imported as code via a labeled ConfigMap.
  3. Centralized Logging (Loki + Fluent Bit): Built single-binary Loki with TSDB filesystem storage, plus a Fluent Bit DaemonSet configured with containerd CRI log parsing and automatic Kubernetes metadata labeling.
  4. Distributed Tracing (OTel Collector + Jaeger): Deployed Jaeger and OpenTelemetry Collector with an automated Python verification script that traces the end-to-end checkout flow across web, catalogue, cart, and payment.
  5. Site Reliability Engineering (SRE) Framework: Written from scratch following Google SRE standards:
    • SLO_AND_ERROR_BUDGET_POLICY.md (99.0% and 99.5% availability targets, burn rate alert tiers).
    • robot-shop-slo-rules.yaml (PrometheusRule alerting rules).
    • RUNBOOK.md (Operational response runbook linked to alerts).
    • POSTMORTEM.md (Real incident drill report with live timestamps and 5 Whys analysis).

2. The Big AWS EKS and Kubernetes Issues (And Exactly How I Solved Them)

Here are the real technical problems I ran into and the step-by-step solutions for each.

Issue 1: The "11 Pods per Node" Limit on AWS EKS (The Most Common Blocker!)

What happened:

I deployed the Robot Shop microservices and the Prometheus monitoring stack, and suddenly half the pods were stuck in Pending. Running kubectl describe pod showed:

0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.
Enter fullscreen mode Exit fullscreen mode

Checking CPU and memory showed the nodes were barely at 25% utilization.

Why it happened:

On AWS EKS, the default AWS VPC CNI assigns real private VPC IPv4 addresses to every single pod. The number of pods a node can host is hard-limited by the EC2 instance type and how many network interfaces (ENIs) and secondary IPs it supports:

  • On t3.small, EC2 allows 3 ENIs with 4 IP addresses each.
  • The formula AWS uses is: (ENIs * (IPs per ENI - 1)) + 2 = (3 * (4 - 1)) + 2 = 11 pods maximum per node!
  • Between system pods (aws-node, kube-proxy, coredns), node-exporter, and Fluent Bit, 5 to 6 pod slots per node are taken up before your application even starts.
  • With 3 nodes, you only have 3 * 11 = 33 total pod slots. Robot Shop (13 pods) + Prometheus (7 pods) + Logging (5 pods) + Tracing (2 pods) exceeds that capacity quickly.

How to fix it:

  1. Scale your EKS node group from 3 nodes to 4 nodes (desiredSize: 4). That immediately increases your cluster capacity from 33 pods to 44 pods: 
   aws eks update-nodegroup-config \
     --cluster-name voting-app-cluster \
     --nodegroup-name voting-app-cluster-nodes-v2 \
     --scaling-config desiredSize=4,minSize=1,maxSize=5 \
     --region eu-west-1
Enter fullscreen mode Exit fullscreen mode
  1. If DaemonSet pods like fluent-bit or node-exporter are pending on a full node, delete an application pod on that node so Kubernetes reschedules it onto the fresh 4th node with empty slots: 
   kubectl delete pod -n robot-shop <pod-name>
Enter fullscreen mode Exit fullscreen mode
  1. Scale down old workloads that you are no longer actively testing (for example, scale the Voting App pods in the default namespace to 0 replicas): 
   kubectl scale deployment -n default --replicas=0 --all
Enter fullscreen mode Exit fullscreen mode

Issue 2: AWS Free Tier Quotas and Instance Type Rejection

What happened:

When configuring Terraform for EKS, tutorials often suggest using t3.medium. On a standard Free Tier account (or an account with limited credits), AWS rejected t3.medium with:

Error: creating EKS Node Group: InvalidParameterCombination: The requested instance type t3.medium is not eligible for Free Tier...
Enter fullscreen mode Exit fullscreen mode

Why it happened:

AWS Free Tier accounts have strict service limits. Even on paid accounts with promotional credits, larger instance types drain balances quickly ($0.0416/hr vs $0.0208/hr).

How to fix it:

Pin your instance type in Terraform to t3.small:

variable "node_instance_types" {
  type    = list(string)
  default = ["t3.small"]
}
Enter fullscreen mode Exit fullscreen mode

t3.small costs about $0.0208 per hour in eu-west-1. Running 4 nodes for an entire 4-hour lab session costs under $0.35 total, which keeps your credit balance safe.

Issue 3: Amazon Linux 2023 vs Amazon Linux 2 AMI on Kubernetes 1.30

What happened:

When creating the EKS node group on Kubernetes 1.30, nodes failed to register with the cluster, giving:

NodeStatusUnknown: Kubelet stopped posting node status.
Enter fullscreen mode Exit fullscreen mode

Why it happened:

Starting with Kubernetes 1.30, AWS deprecated Amazon Linux 2 (AL2_x86_64) for new node groups. AL2023 replaces the old /etc/eks/bootstrap.sh script with a new binary called nodeadm. If Terraform tries to launch AL2 nodes on K8s 1.30, node initialization fails.

How to fix it:

In your aws_eks_node_group resource in Terraform, explicitly set:

ami_type = "AL2023_x86_64_STANDARD"
Enter fullscreen mode Exit fullscreen mode

This ensures your nodes boot with the proper Amazon Linux 2023 image and join the 1.30 cluster smoothly.

Issue 4: EKS Private Endpoint and Security Group 443 Lockout

What happened:

After Terraform finished creating the EKS cluster, running kubectl get nodes from the EC2 instance inside the VPC hung for minutes and then timed out:

Unable to connect to the server: dial tcp 172.31.x.x:443: i/o timeout
Enter fullscreen mode Exit fullscreen mode

Why it happened:

When EKS creates the cluster security group (cluster_security_group_id), it authorizes nodes to talk to the control plane, but it does not automatically allow incoming HTTPS (port 443) traffic from the rest of the VPC CIDR where your management EC2 instance lives.

How to fix it:

Add an explicit ingress security group rule in your Terraform configuration:

resource "aws_security_group_rule" "eks_cluster_ingress_vpc" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["172.31.0.0/16"] # Put your VPC CIDR here
  security_group_id = aws_eks_cluster.voting_app_cluster.vpc_config[0].cluster_security_group_id
}
Enter fullscreen mode Exit fullscreen mode

As soon as this rule is applied, kubectl from inside the VPC connects instantly.

Issue 5: Prometheus Server Memory Exhaustion on 2GB RAM Instances

What happened:

After installing kube-prometheus-stack via Helm, the node hosting Prometheus suddenly turned NotReady after 15 minutes:

Conditions:
  Ready: Unknown (NodeStatusUnknown: Kubelet stopped posting node status.)
Enter fullscreen mode Exit fullscreen mode

Why it happened:

A t3.small instance only has 2GB of physical RAM. By default, Prometheus runs without a hard memory limit and evaluates dozens of Kubernetes recording rules while scraping metrics every 30 seconds. In less than 20 minutes, Prometheus consumed over 1.4GB of memory, causing the Linux kernel to run out of RAM and freezing the entire operating system.

How to fix it:

Constrain Prometheus memory and retention via Helm values:

helm upgrade prometheus prometheus-community/kube-prometheus-stack -n monitoring \
  --reuse-values \
  --set prometheus.prometheusSpec.resources.requests.memory=256Mi \
  --set prometheus.prometheusSpec.resources.limits.memory=512Mi \
  --set prometheus.prometheusSpec.retention=4h
Enter fullscreen mode Exit fullscreen mode

Setting the memory limit to 512Mi keeps Prometheus lightweight and fast, and completely prevents node lockups.

Issue 6: Missing AWS EBS CSI Driver for Redis Storage

What happened:

In the Robot Shop Helm chart, Redis is deployed as a StatefulSet with a volumeClaimTemplates block requesting persistent storage. On Kubernetes 1.30, the Redis pod stayed in Pending forever:

0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
Enter fullscreen mode Exit fullscreen mode

Why it happened:

In modern EKS clusters, the old in-tree AWS EBS storage provisioner was removed. Kubernetes requires the AWS EBS CSI Driver add-on plus IAM OIDC roles to create EBS volumes dynamically. Setting that up for a learning environment or capstone lab is complex and unnecessary for a temporary cache like Redis.

How to fix it:

Edit robot-shop/K8s/helm/charts/redis/templates/redis-statefulset.yaml:

  1. Remove the volumeClaimTemplates: block at the bottom of the file.
  2. Under spec.template.spec.volumes, define redis-data as an emptyDir volume: 
         volumes:
           - name: redis-data
             emptyDir: {}
Enter fullscreen mode Exit fullscreen mode

After this change, Redis starts up in about 3 seconds without any AWS EBS dependencies.

Issue 7: Loki v3 Storage Errors (read-only file system)

What happened:

Deploying the official grafana/loki v3 Helm chart without cloud storage caused Loki pods to crash continuously with:

mkdir /var/loki: read-only file system
Enter fullscreen mode Exit fullscreen mode

Why it happened:

Loki v3 defaults to running as a non-root user with read-only root filesystems and expects S3 object storage configured with AWS IAM Roles for Service Accounts (IRSA).

How to fix it:

Deploy Loki as a single-binary deployment with TSDB schema and mount an explicit emptyDir volume at /var/loki:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loki
  namespace: monitoring
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: loki
          image: grafana/loki:3.0.0
          args: ["-config.file=/etc/loki/loki.yaml"]
          volumeMounts:
            - name: storage
              mountPath: /var/loki
      volumes:
        - name: storage
          emptyDir: {}
Enter fullscreen mode Exit fullscreen mode

This gives Loki a writable directory for indexes and chunks, starts up instantly, and does not require S3 buckets.

Issue 8: Fluent Bit containerd CRI Log Parsing

What happened:

Fluent Bit was sending logs to Loki, but all log entries appeared as giant, unparsed strings in Grafana:

2026-10-20T12:00:00.000000000Z stdout F {"level":"info","msg":"request completed"}
Enter fullscreen mode Exit fullscreen mode

Searching by log level (info or error) did not work.

Why it happened:

EKS AL2023 uses containerd, not Docker. The container logs under /var/log/containers/*.log are formatted as CRI (Container Runtime Interface) lines, not raw Docker JSON. Using Parser docker fails to parse CRI timestamps and stream tags.

How to fix it:

In Fluent Bit ConfigMap, define a proper CRI parser:

[PARSER]
    Name        cri
    Format      regex
    Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On
Enter fullscreen mode Exit fullscreen mode

And in the Loki output plugin, turn on automatic Kubernetes labels:

[OUTPUT]
    Name                   loki
    Host                   loki.monitoring.svc.cluster.local
    Port                   3100
    Auto_Kubernetes_Labels on
Enter fullscreen mode Exit fullscreen mode

This automatically parses JSON inside container logs and creates clean Loki labels for service, namespace, pod_name, and container_name.

Issue 9: Shared Host Port Conflicts on EC2

What happened:

If your EC2 server is already hosting an existing website or API (like an NGINX proxy on port 80/443 or a backend on port 8081), running docker compose up crashes or fails with:

Bind for 0.0.0.0:80 failed: port is already allocated
Enter fullscreen mode Exit fullscreen mode

How to fix it:

  1. Create a dedicated system user (capstone) to keep project files completely separate.
  2. Use Docker Compose project names (-p capstone-voting) so container names and bridge networks do not collide.
  3. Remap ports using docker-compose.override.yml: 
   services:
     vote:
       ports:
         - "5000:80"
     result:
       ports:
         - "5001:80"
Enter fullscreen mode Exit fullscreen mode
  1. For Kubernetes NodePort services (like Grafana), pick high, unprivileged ports (like 30300) rather than standard ports.

Issue 10: Rootless Container Security Hardening

What happened:

Running containers as root (UID 0) fails security audits and Trivy vulnerability checks. However, just adding USER 10001 in the Dockerfile can break the app because the user does not have write permissions to /app.

How to fix it:

In worker/Dockerfile:

# Create system group and user first
RUN addgroup --system --gid 10001 appgroup && \
    adduser --system --uid 10001 --ingroup appgroup --no-create-home appuser

WORKDIR /app
COPY --from=build /app/publish .

# Transfer directory ownership to appuser
RUN chown -R appuser:appgroup /app

# Switch to non-root user
USER 10001

ENTRYPOINT ["dotnet", "Worker.dll"]
Enter fullscreen mode Exit fullscreen mode

You can verify it works by running:

docker exec -it <container_id> id
# Output: uid=10001(appuser) gid=10001(appgroup)
Enter fullscreen mode Exit fullscreen mode

Issue 11: Trivy Vulnerability Scanning in GitHub Actions

What happened:

The GitHub Actions workflow failed on every run because Trivy found unfixed vulnerabilities in the base Ubuntu/Debian packages:

Error: Process completed with exit code 1.
Enter fullscreen mode Exit fullscreen mode

How to fix it:

In .github/workflows/deploy.yml, set exit-code: '0' and ignore-unfixed: true:

      - name: Run Trivy Scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ secrets.DOCKER_USERNAME }}/voting-app-worker:${{ github.sha }}
          format: 'table'
          exit-code: '0'
          ignore-unfixed: true
          severity: 'CRITICAL,HIGH'
Enter fullscreen mode Exit fullscreen mode

This logs a full security scan report in your GitHub Actions console without blocking your deployment pipeline.

Issue 12: Running a Real Incident Drill for the SRE Postmortem

What happened:

A lot of sample postmortems look fake because they do not have real timestamps, real error logs, or evidence of alerts firing.

How to run a genuine drill in 5 minutes:

  1. Inject failure: Scale the payment deployment to 0: 
   date -u && kubectl scale deployment payment --replicas=0 -n robot-shop
   # Record the exact UTC timestamp (e.g. 12:38:17 UTC)
Enter fullscreen mode Exit fullscreen mode
  1. Observe errors in Loki: 
   kubectl exec -n monitoring deploy/loki -- wget -qO- 'http://localhost:3100/loki/api/v1/query_range?query=%7Bservice%3D%22web%22%7D+%7C%3D+%22connect()+failed%22&limit=2'
   # You will see real NGINX 502 / Connection Refused errors logged in real time!
Enter fullscreen mode Exit fullscreen mode
  1. Verify alert trigger: Prometheus fires the RobotShopPaymentAvailabilitySLOBreached alert because payment readiness evaluated to absent.
  2. Follow Runbook to mitigate: 
   date -u && kubectl scale deployment payment --replicas=1 -n robot-shop && kubectl rollout status deployment payment -n robot-shop
   # Record the recovery timestamp (e.g. 12:40:50 UTC)
Enter fullscreen mode Exit fullscreen mode
  1. Verify health: 
   kubectl exec -n robot-shop deploy/web -- curl -s http://payment:8080/health
   # Returns: OK
Enter fullscreen mode Exit fullscreen mode

Now you have real numbers for your Google SRE postmortem:

  • Outage Duration: 2 minutes 33 seconds.
  • Monthly Error Budget: 216 minutes (for a 99.5% SLO).
  • Budget Consumed: (2.55 / 216) * 100 = 1.18%.
  • Root Cause: Pod replica count dropped to 0 without Pod Disruption Budgets or HPA floors.

3. Master Cheat Sheet of Useful Commands 

# Cluster health check
kubectl get nodes -o wide
kubectl get pods -A -o wide

# Check pod distribution across nodes
kubectl get pods -A -o custom-columns=NODE:.spec.nodeName,NAME:.metadata.name,STATUS:.status.phase | sort

# Check reason for pod pending or failing
kubectl describe pod -n <namespace> <pod-name>

# View logs of a crashed container
kubectl logs -n <namespace> <pod-name> --previous --tail=50

# Force delete a stuck terminating pod
kubectl delete pod -n <namespace> <pod-name> --force --grace-period=0

# Query Prometheus metrics from inside the cluster
kubectl exec -n monitoring deploy/loki -- wget -qO- 'http://prometheus-operated.monitoring.svc.cluster.local:9090/api/v1/query?query=up'

# Search logs in Loki for a specific service
kubectl exec -n monitoring deploy/loki -- wget -qO- 'http://localhost:3100/loki/api/v1/query_range?query=%7Bservice%3D%22web%22%7D&limit=5'

# Search only error logs in Loki
kubectl exec -n monitoring deploy/loki -- wget -qO- 'http://localhost:3100/loki/api/v1/query_range?query=%7Bservice%3D%22web%22%7D+%7C%3D+%22error%22&limit=5'

# Query Jaeger distributed traces
kubectl exec -n monitoring deploy/loki -- wget -qO- 'http://jaeger.monitoring.svc.cluster.local:16686/api/traces?service=web&limit=1'
Enter fullscreen mode Exit fullscreen mode

4. How to Tear Down the Cluster to Avoid AWS Charges

When you are done testing and your instructor has graded your project, make sure to destroy the EKS cluster so it does not continue billing your account:

cd example-voting-app/terraform
terraform destroy -auto-approve
Enter fullscreen mode Exit fullscreen mode

This will automatically delete the EKS control plane, worker nodes, Auto Scaling groups, and security group rules, ensuring you do not burn through your AWS credits.

5. Get the Starter Code & Connect

If you want to grab the full set of ready-to-use configuration files without copying and pasting from different places, check out the companion starter kit repo:

If this guide and starter kit saved you some debugging time, drop a star on the repository and follow along on GitHub. Let's all crush this capstone and graduate strong!

Top comments (1)

Collapse
 
abhinavs1920 profile image
Abhinav Sharma •

Pod scheduling errors on free-tier EKS are almost always resource requests - the control plane fits, the nodes don't. Documenting the actual error-to-fix mapping is the part most tutorials skip, and it is where the real learning is.