DEV Community: Mustkhim Inamdar

Troubleshooting Real-World AWS EKS Issues in Production

Mustkhim Inamdar — Fri, 11 Jul 2025 04:04:37 +0000

Amazon EKS makes it easier to run Kubernetes workloads in the cloud but as any platform engineer knows, production grade reliability still demands deep visibility, sound architecture, and well drilled troubleshooting.

In this post, I’ll walk you through some real-world EKS incidents I’ve personally resolved. You'll find:

Root causes (RCA)
Troubleshooting steps
Fixes and lessons learned
Diagrams and code

Let’s get into it 👇

1. Node in NotReady State

Symptoms:

kubectl get nodes → shows NotReady
Pods evicted or stuck
High node disk usage or kubelet crash

Fix:

# Check disk space
df -h

# Clear logs
sudo truncate -s 0 /var/log/containers/*.log

# Restart kubelet
sudo systemctl restart kubelet

# Replace node
kubectl drain <node> --ignore-daemonsets --delete-local-data
kubectl delete node <node>

🌐 Visual: Node in NotReady State

+------------------------+
|      EKS Node          |
|------------------------|
|  Kubelet process       |
|  Disk usage &gt; 85%      |
|  Memory pressure       |
+-----------+------------+
            |
            | Heartbeat to API server fails
            v
+------------------------+
| Kubernetes Control Plane|
|------------------------|
| Node marked NotReady    |
| Events generated        |
+-----------+------------+
            |
            | Admin runs diagnostics
            v
+------------------------+
| Resolution actions      |
| - Clear disk/logs       |
| - Restart kubelet       |
| - Drain/replace node    |
+------------------------+

2. LoadBalancer Service Stuck in Pending

Symptoms:

kubectl get svc → EXTERNAL-IP = <pending>
No ELB visible in AWS Console

Fix:

# Tag subnets
aws ec2 create-tags --resources <subnet-id> \
  --tags Key=kubernetes.io/cluster/<cluster>,Value=shared

# Install AWS Load Balancer Controller
helm repo add eks https://aws.github.io/eks-charts
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=<cluster-name> \
  --set serviceAccount.name=aws-load-balancer-controller \
  -n kube-system

🌐 Visual: EKS LoadBalancer Flow

+--------------------------+
|  Kubernetes Service (LB) |
|  Type: LoadBalancer      |
|  Exposes app externally  |
+-----------+--------------+
            |
            | Triggers AWS ELB provisioning
            v
+--------------------------+
|   aws-load-balancer-     |
|       controller         |
| (Installed via Helm)     |
+-----------+--------------+
            |
            | Creates ELB in AWS
            v
+--------------------------+
|     AWS Elastic LB       |
| - Internet-facing ELB    |
| - Listens on 80/443      |
+-----------+--------------+
            |
            | Forwards traffic to EKS nodes
            v
+--------------------------+
|     EKS Worker Nodes     |
| - Backed by Target Group |
| - Runs app pods          |
+--------------------------+

3. Pods in CrashLoopBackOff

Symptoms:

Application crashes repeatedly
kubectl describe pod shows BackOff
Logs show stack traces or missing configs

Fix:

# Check logs
kubectl logs <pod> -c <container>

# Edit deployment and fix
kubectl edit deployment <deployment-name>

🌐 Visual: CrashLoopBackOff Lifecycle

+------------------------+
|      Kubernetes        |
|     Deployment/Job     |
+----------+-------------+
           |
           | Schedules Pod
           v
+------------------------+
|        Pod Starts      |
|  init containers run   |
+----------+-------------+
           |
           | Starts Main Container
           v
+------------------------+
|  Container Crashes     |  &lt;---- App bug, bad config, secret missing
|  Exit Code ≠ 0         |
+----------+-------------+
           |
           | Kubernetes Restarts Pod
           v
+------------------------+
| CrashLoopBackOff Timer |
+------------------------+

4. API Server Latency & 5xx Errors

Symptoms:

kubectl commands slow or fail
High API server metrics in CloudWatch
Prometheus or controllers hammering API

Fix:

# Example Prometheus config
global:
  scrape_interval: 60s

Batch CRD updates
Add jitter/delay in loops

🌐 Visual: API Server Under Stress

+-----------------------+
|      Clients          |
| - kubectl, CI/CD      |
| - Prometheus scrapes  |
| - Controllers         |
+-----------+-----------+
            |
            | High-frequency requests
            v
+----------------------------+
|     Kubernetes API Server |
| - Latency / 5xx errors     |
| - Queued requests          |
+-----------+---------------+
            |
            v
+-----------------------------+
|            etcd             |
| - Slow writes/timeouts      |
+-----------------------------+

5. DNS Resolution Fails in Pod

Symptoms:

App can't resolve DNS (e.g., can't reach mydb.default.svc)
ping or nslookup fails inside pod
resolv.conf points to broken CoreDNS

Fix:

kubectl rollout restart deployment coredns -n kube-system

# Edit CoreDNS config if needed
kubectl edit configmap coredns -n kube-system

🌐 Visual: DNS Failure Inside Pod

+--------------------+        +---------------------+
|   Pod in EKS Node  |  --&gt;   |    CoreDNS Pod      |
| /etc/resolv.conf   |        | Resolves DNS queries|
| nameserver 10.x.x.x|        +----------+----------+
+--------+-----------+                   |
         |                               v
         |                     +---------------------+
         |                     |   VPC Network Layer  |
         |                     | - NACL / SG rules    |
         +--------------------&gt;+---------------------+
                                |
                                v
                        DNS Query Fails
                      (Timeout or No Response)
                                |
                                v
                    App Failure / Connection Error

💡 Final Takeaways

Always run postmortems
Ensure IAM/SG/Subnet configs are in place
Monitor metrics, logs, and event streams
Automate node drain and self-healing

About the Author

Mustkhim Inamdar
Cloud-Native DevOps Architect | Platform Engineer | CI/CD Specialist
Passionate about automation, scalability, and next-gen tooling. With years of experience across Big Data, Cloud Operations (AWS), CI/CD, and DevOps for automotive systems, I’ve delivered robust solutions using tools like Terraform, Jenkins, Kubernetes, LDRA, Polyspace, MATLAB/Simulink, and more.

I love exploring emerging tech like GitOps, MLOps, and Generative AI, and sharing practical insights from real-world projects.
🔗 LinkedIn
🔗 GitHub

Do bookmark ⭐ if you found this helpful. Comment below if you want me to share full runbooks or reusable Terraform modules I’ve built for EKS production clusters.

Jenkins in Production: Real Issues, RCA, and Fixes That Actually Work

Mustkhim Inamdar — Tue, 01 Jul 2025 11:35:15 +0000

This isn’t theory. This is a real production issue I faced with Jenkins documented with actual RCA, the troubleshooting I followed, and how I fixed and hardened the pipeline after recovery.

1. Jenkins Master Became Unresponsive During Peak Hours

What happened:

Jenkins UI crashed
Builds queued indefinitely
Engineers across multiple teams blocked

Duration of impact:

~2 hours

Root Cause:

JVM heap space exhausted → OutOfMemoryError
Disk usage at 100% → no cleanup of old builds/artifacts
SCM hooks and Git polling overwhelmed the executor queue
No workspace cleanup on matrix builds

Troubleshooting

tail -f /var/log/jenkins/jenkins.log
jcmd <pid> GC.heap_info
df -h
htop

Killed large zombie processes
Cleared build directories
Restarted Jenkins after freeing up memory

✅ Fixes Applied

JVM Memory Configuration

export JENKINS_JAVA_OPTIONS="-Xms2g -Xmx4g"

Pipeline Cleanup & Discarder

options {
  buildDiscarder(logRotator(numToKeepStr: '10'))
}

post {
  always {
    cleanWs()
  }
}

Log Rotation & Backup

tar -czf jenkins_backup_$(date +%F).tar.gz /var/lib/jenkins

SCM/Webhook Control

GitHub webhook throttled
Added quiet periods and throttle plugin

2. Jenkins Agent Disconnecting Mid-Build

What happened:

Builds failed halfway through execution
Logs were incomplete
Rebuilds triggered, wasting compute

Root Cause:

SSH/JNLP connection dropped due to firewall timeout
Cloud auto-scaling agents terminated mid-job
No agent lifecycle hooks defined

Troubleshooting

Checked jenkins.log and agent logs
Verified cloud termination settings
Monitored for memory/cpu bottlenecks on agents

✅Fixes Applied

Enable TCP Keep-Alive

ServerAliveInterval 60
ServerAliveCountMax 5

Cloud Agent Protection

Graceful shutdown scripts
Increased idle timeout before scale-in
Only terminate idle agents with no active job

3. Jenkins Master UI Was Freezing Frequently

What happened:

Jenkins dashboard became sluggish
Admin actions timed out
Pipelines remained queued

Root Cause:

Scripted pipelines executing heavy shell operations on master
Plugins with memory leaks
Too many concurrent builds on master thread

Troubleshooting

Analyzed /metrics endpoint
Monitored heap and thread dumps
Disabled high-impact plugins

✅Fixes Applied

Move All Jobs Off Master

Set “Restrict where this project runs”
Enforce master isolation

Switch to Declarative Pipelines

Reduced memory footprint
Improved readability and safety

Enable Monitoring

Installed Metrics Plugin
Integrated with Prometheus + Grafana

Secrets Leaked into Logs and Artifacts

What happened:

Tokens and .pem files showed up in Jenkins logs
.env files archived as pipeline artifacts
Security audit failed

Root Cause:

Use of echo $TOKEN inside sh blocks
No use of withCredentials wrapper
Logs not masked automatically

Troubleshooting

Scanned logs using keywords like AKIA, BEGIN RSA, etc.
Reviewed artifact contents for secrets
Reviewed pipeline code across teams

✅ Fixes Applied

Mask Secrets

withCredentials([string(credentialsId: 'secret-token', variable: 'TOKEN')]) {
  sh 'curl -H "Authorization: Bearer $TOKEN" https://secure-api/'
}

Block Sensitive Artifact Upload

steps {
  sh '''
    if grep -r 'AKIA' ./build; then exit 1; fi
  '''
}

Enable Secret Scanning Tools

Integrated GitLeaks
Pre-check builds for secret patterns

Jenkins Plugin Incompatibility After Upgrade

What happened:

Jenkins failed to start after a routine upgrade
Multiple jobs crashed due to missing plugin dependencies
UI elements broke, pipelines wouldn't compile

Root Cause:

Plugin versions upgraded without checking compatibility
Jenkins core version jumped ahead
Deprecated scripted pipelines using outdated plugin APIs

Troubleshooting

Accessed Jenkins in safe mode
Checked /var/lib/jenkins/plugins
Rolled back version via backup restore

✅ Fixes Applied

Version Lock with plugins.txt

git:4.11.5
workflow-aggregator:2.6
credentials:2.6.1

Test Updates on Staging First

Jenkins Docker image with pinned plugins
Automated plugin diff validation via jenkins-plugin-cli

Upgrade Policy Aligned with LTS Cycle

📋 Summary Table: RCA & Fixes

Issue	Root Cause	Fix Applied
Master Unresponsive	Heap/Disk full, SCM flooding	Memory tuning, cleanup, webhook control
Agent Disconnects	Network timeout, auto-scale kills	Keep-alive, lifecycle hooks
UI Freezing	Master overload, heavy plugins	Pipeline refactor, monitoring
Secrets in Logs	Unsafe usage of shell/env	withCredentials, scanning
Plugin Failures	Incompatible versions	Pin versions, test on staging

✅ Jenkins Production Readiness Checklist

[x] JVM heap and thread monitoring
[x] Log/artifact cleanup via pipeline config
[x] Declarative pipelines with clean stages
[x] Secrets masked and scanned
[x] Plugin versions pinned in code
[x] Staging Jenkins for dry runs
[x] Backup + disaster recovery tested monthly

Final Take

Jenkins is a battle-tested CI/CD engine but left unchecked, it can become fragile and costly in production. These 5 real-world issues cost teams hours, if not days. But they also taught us how to:

Think of Jenkins like core infrastructure
Use IaC principles to control configuration
Automate hygiene and disaster recovery

About the Author

Mustkhim Inamdar
Cloud-Native DevOps Architect | Platform Engineer | CI/CD Specialist
Passionate about automation, scalability, and next-gen tooling. With years of experience across Big Data, Cloud Operations (AWS), CI/CD, and DevOps for automotive systems, I’ve delivered robust solutions using tools like Terraform, Jenkins, Kubernetes, LDRA, Polyspace, MATLAB/Simulink, and more.

I love exploring emerging tech like GitOps, MLOps, and Generative AI, and sharing practical insights from real-world projects.

📬 Let’s connect:
🔗 LinkedIn
📘 GitHub
🧠 Blog series on DevOps + AI coming soon!

💬 Got your own Jenkins horror story?
Drop it in the comments or DM me on LinkedIn. Let’s learn from each other’s scars and build resilient CI/CD systems.

Building Scalable Modular AWS Infrastructure with Terraform (IaC)

Mustkhim Inamdar — Mon, 23 Jun 2025 09:57:19 +0000

by M Inamdar

👋 Hey there! I’m Mustkhim Inamdar, a Cloud-Native DevOps Architect passionate about automation, scalability, and next-gen tooling. With 9+ years of experience across Big Data, Cloud Operations (AWS), CI/CD, and DevOps for automotive systems, I’ve delivered robust solutions using tools like Terraform, Jenkins, Kubernetes, LDRA, Polyspace, MATLAB/Simulink, and more.

I love exploring emerging tech like GitOps, MLOps, and GenAI, and sharing practical insights from real-world projects. Let’s dive into the world of DevOps, cloud, and automation together!

Modern cloud infrastructure demands automation, scalability, and maintainability and that’s exactly where Infrastructure as Code (IaC) shines. But as your infrastructure grows, maintaining hundreds of lines of Terraform in a single file becomes chaotic.

That's why I built this:
A clean, modular, and production-grade Terraform repository that can deploy scalable AWS infrastructure across multiple environments.

This article is a deep dive into the repo:
iac-aws-modular-infra

Why This Project Exists

Most Terraform tutorials and starter templates are overly simple or don’t reflect real-world complexity. This project is built for engineers and teams who need:

✅ Modular, reusable infrastructure components
✅ Multi-environment separation (e.g., dev, prod)
✅ Remote state management
✅ Cleaner code structure and CI/CD-ready setup
✅ A launchpad for integrating EKS, RDS, etc.

This repo follows the best practices recommended by HashiCorp and the Terraform community.

What’s Inside the Repo?

This Terraform repo includes the following modular AWS resources:

Module	Description
`vpc`	Creates custom VPCs, subnets, route tables, IGW, etc.
`ec2`	Deploys EC2 instances inside public/private subnets
`s3`	Creates S3 buckets with custom configuration
`sg`	Custom security groups for EC2, load balancers, etc.
`iam`	IAM roles and policies (basic version included)
`cloudwatch`	Adds alarms and monitoring for EC2 or custom metrics

Directory Layout

iac-aws-modular-infra/
│
├── backend/                # Remote state config using S3
│   └── s3_backend.tf
│
├── environments/           # Separated environments
│   ├── dev/
│   │   └── main.tf
│   └── prod/
│
├── modules/                # All infrastructure modules
│   ├── vpc/
│   ├── ec2/
│   ├── s3/
│   ├── sg/
│   ├── iam/
│   └── cloudwatch/
│
└── README.md

How to Deploy

1. Clone the Repo

Clone the infrastructure code to your local machine and navigate to the desired environment (e.g., dev).

git clone https://github.com/M-Inamdar/iac-aws-modular-infra.git
cd iac-aws-modular-infra/environments/dev

2. Create `terraform.tfvars`

Define environment-specific values for input variables. This file keeps your configuration modular and reusable.

region               = "ap-south-1"
vpc_cidr_block       = "10.0.0.0/16"
public_subnet_cidrs  = ["10.0.1.0/24"]
private_subnet_cidrs = ["10.0.2.0/24"]
instance_type        = "t2.micro"
ami_id               = "ami-0abcdef1234567890"

Tip: Add this file to .gitignore to avoid committing sensitive or environment-specific data.

3. Initialize Terraform

Set up the working directory and download the required provider plugins and modules.

terraform init

4. Plan & Apply

Preview the changes with plan and then provision the infrastructure using apply.

terraform plan

terraform apply

✅ You’ll be prompted to confirm before resources are created.

5. Optional: Destroy Infrastructure

Use this to tear down all provisioned resources if no longer needed.

terraform destroy

Caution: This will remove all resources. Double-check before running.

Remote State Management

Terraform uses an S3 bucket to manage state centrally for team collaboration and history tracking.

Example backend config in backend/s3_backend.tf:

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "dev/terraform.tfstate"
    region = "ap-south-1"
  }
}

Tip: Enable DynamoDB state locking to avoid concurrent apply conflicts:

dynamodb_table = "terraform-locks"

Additional Suggestions

Modular Code: Reuse modules from the modules/ directory across different environments (e.g., dev, staging, prod).
Secrets Management: Avoid hardcoding credentials. Use environment variables or tools like AWS Secrets Manager or SSM Parameter Store.
Validation: Run terraform validate before applying to catch config errors.
Automation: Integrate with CI/CD pipelines (e.g., GitHub Actions, Jenkins) for consistent deployment.
Documentation: Maintain a README per environment or module for clarity.

Design Philosophy

“Infrastructure should be treated as code and that code should be clean, reusable, and easy to test.”

I designed this repo with the following principles:

✅ Modularity: Each AWS resource lives in its own module
✅ Reusability: Modules can be called from any environment
✅ Clarity: Variables and outputs are explicitly defined
✅ Separation of concerns: Code for prod/dev stays isolated
✅ Cloud-readiness: Backend is set up for remote storage

What’s Next?

Here’s what I’m planning to add next:

Terraform GitHub Actions for CI/CD
Cost estimates with Infracost
State locking via DynamoDB
Terratest-based module unit tests
EKS-ready VPC modules

TL;DR

Modular Terraform setup ✅
Real-world AWS patterns ✅
Remote state and multi-env support ✅
Ready for scaling and CI/CD ✅

🤝 Contributing

Feel free to fork the repo, raise issues, or open pull requests. Feedback and contributions are always welcome!

If you use this in your projects, I’d love to hear about it drop a comment or star ⭐ the repo iac-aws-modular-infra

Happy automating!

💬 Got questions or stuck somewhere?

Feel free to drop a comment below or DM me on LinkedIn.

I’m always happy to help!

DEV Community: Mustkhim Inamdar

Troubleshooting Real-World AWS EKS Issues in Production

Jenkins in Production: Real Issues, RCA, and Fixes That Actually Work

1. Jenkins Master Became Unresponsive During Peak Hours

What happened:

Duration of impact:

Root Cause:

Troubleshooting

✅ Fixes Applied

2. Jenkins Agent Disconnecting Mid-Build

What happened:

Root Cause:

Troubleshooting

✅Fixes Applied

3. Jenkins Master UI Was Freezing Frequently

What happened:

Root Cause:

Troubleshooting

✅Fixes Applied

Secrets Leaked into Logs and Artifacts

What happened:

Root Cause:

Troubleshooting

✅ Fixes Applied

Jenkins Plugin Incompatibility After Upgrade

What happened:

Root Cause:

Troubleshooting

✅ Fixes Applied

📋 Summary Table: RCA & Fixes

✅ Jenkins Production Readiness Checklist

Final Take

About the Author

Building Scalable Modular AWS Infrastructure with Terraform (IaC)

Why This Project Exists

What’s Inside the Repo?

Directory Layout

How to Deploy

1. Clone the Repo

2. Create terraform.tfvars

3. Initialize Terraform

4. Plan & Apply

5. Optional: Destroy Infrastructure

Remote State Management

Additional Suggestions

Design Philosophy

What’s Next?

TL;DR

🤝 Contributing

2. Create `terraform.tfvars`