DEV Community: Nowsath

Deploying a Node.js E-Commerce API with Amazon ECS Express Mode: A Revolution in Container Deployment

Nowsath — Sun, 18 Jan 2026 13:55:58 +0000

The traditional process of deploying containerized applications to AWS has long been a multi-step journey involving manual configuration of load balancers, auto-scaling groups, VPCs, security groups, and domain mappings. For developers who simply want to get their applications running in production, this complexity can be frustrating and time-consuming.

Enter Amazon ECS Express Mode, announced at AWS re:Invent 2025. This game-changing feature transforms container deployment from a complex, multi-hour configuration task into a single command that handles everything automatically. In this article, we'll walk through deploying a real-world e-commerce product API using ECS Express Mode, demonstrating just how simple cloud deployment has become.

The Use Case: Product Catalog API

Let's build and deploy a realistic product catalog API for an e-commerce platform. This API will handle:

Product listing and search
Product details retrieval
Inventory management
High-traffic scenarios (typical for e-commerce during sales events)

This is exactly the kind of application that benefits from containerization, auto-scaling, and load balancing features that traditionally required extensive AWS configuration.

Step 1: Building the Application

First, let's create our Node.js application. Here's a simplified but functional product API:

app.js:

const express = require('express');
const app = express();
const port = process.env.PORT || 3000;

app.use(express.json());

// Mock product database
let products = [
  { id: 1, name: "Wireless Headphones", price: 89.99, stock: 150 },
  { id: 2, name: "Smart Watch", price: 249.99, stock: 75 },
  { id: 3, name: "Laptop Stand", price: 45.50, stock: 200 },
  { id: 4, name: "USB-C Hub", price: 39.99, stock: 300 }
];

// Health check endpoint (important for load balancer)
app.get('/health', (req, res) => {
  res.status(200).json({ status: 'healthy' });
});

// Get all products
app.get('/api/products', (req, res) => {
  res.json({ products, total: products.length });
});

// Get product by ID
app.get('/api/products/:id', (req, res) => {
  const product = products.find(p => p.id === parseInt(req.params.id));
  if (!product) {
    return res.status(404).json({ error: 'Product not found' });
  }
  res.json(product);
});

// Update inventory
app.patch('/api/products/:id/inventory', (req, res) => {
  const product = products.find(p => p.id === parseInt(req.params.id));
  if (!product) {
    return res.status(404).json({ error: 'Product not found' });
  }
  product.stock = req.body.stock;
  res.json(product);
});

app.listen(port, () => {
  console.log(`Product API running on port ${port}`);
});

package.json:

{
  "name": "product-api",
  "version": "1.0.0",
  "main": "app.js",
  "scripts": {
    "start": "node app.js"
  },
  "dependencies": {
    "express": "^4.18.2"
  }
}

Step 2: Containerizing the Application

Create a Dockerfile to package our application:

Dockerfile:

FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm install --production

COPY app.js ./

EXPOSE 3000

CMD ["npm", "start"]

Step 3: The Magic - Deploying with ECS Express Mode

Here's where the revolution happens. Instead of spending hours configuring AWS resources, we deploy with a single command:

aws ecs express deploy \
  --name product-api \
  --image product-api:latest \
  --port 3000 \
  --domain api.mystore.com \
  --auto-scale-min 2 \
  --auto-scale-max 10 \
  --health-check-path /health

That's it. One command.

What Happens Behind the Scenes?

When you run this single command, ECS Express Mode automatically:

Creates an ECR repository and pushes your Docker image
Provisions a VPC with public and private subnets across multiple availability zones
Sets up an Application Load Balancer with health checks pointing to /health
Configures security groups allowing traffic on port 3000 from the load balancer
Creates an ECS cluster with Fargate capacity
Deploys your containers with the specified minimum of 2 instances
Configures auto-scaling to handle traffic spikes (up to 10 instances)
Maps your custom domain (api.mystore.com) with SSL/TLS certificate provisioning
Sets up CloudWatch logging for monitoring and debugging

All of this happens automatically, typically completing in under 5 minutes.

Step 4: Verifying the Deployment

Once deployment completes, you can immediately test your API:

# Get all products
curl https://api.mystore.com/api/products

# Get specific product
curl https://api.mystore.com/api/products/1

# Update inventory
curl -X PATCH https://api.mystore.com/api/products/1/inventory \
  -H "Content-Type: application/json" \
  -d '{"stock": 125}'

Real-World Benefits

Time Savings
Traditional ECS deployment: 2-4 hours of configuration
ECS Express Mode deployment: 5 minutes

Cost Optimization
The auto-scaling configuration means you only pay for what you need. During low-traffic periods, you run 2 containers. During a flash sale, the system automatically scales to 10 containers and back down when traffic subsides.

Production-Ready from Day One
You get enterprise-grade features without manual configuration including high availability across multiple availability zones, automatic health checks and container replacement, SSL/TLS encryption, distributed load balancing, and comprehensive monitoring and logging.

Monitoring and Management

View your deployment status:

aws ecs express status --name product-api

Check logs:

aws ecs express logs --name product-api --follow

Update the deployment with a new version:

aws ecs express deploy \
  --name product-api \
  --image product-api:v2.0 \
  --update

When to Use ECS Express Mode

ECS Express Mode is ideal for situations where you want to deploy containerized applications quickly without infrastructure complexity, need production-grade features (load balancing, auto-scaling) without manual setup, are prototyping or building MVPs or are migrating from simpler platforms like Heroku to AWS. You want to focus on application code, not infrastructure.

It might not be the best fit if you need highly customized networking configurations, have complex multi-service architectures requiring service mesh or need to integrate with existing custom VPC configurations.

Conclusion

Amazon ECS Express Mode represents a fundamental shift in how we deploy containerized applications to AWS. By abstracting away infrastructure complexity while maintaining enterprise-grade capabilities, it enables developers to focus on what truly matters-building great applications.

Our product API deployment demonstrates that you can go from code to production-ready, auto-scaling, load-balanced infrastructure with a single command. This is the future of cloud deployment: simple, powerful and developer-friendly.

Whether you're a startup looking to move fast or an enterprise team wanting to reduce operational overhead, ECS Express Mode offers a compelling path to modern containerized deployments on AWS.

Try It Yourself

Ready to deploy your own application with ECS Express Mode? Here are the prerequisites:

AWS CLI version 2.15 or later
Docker installed locally
An AWS account with appropriate permissions
A container image ready to deploy

Start with the simple command shown in this article and experience the future of container deployment today.

Security Considerations for Multi-Cluster Cloud Architecture (HA EKS with Databases)

Nowsath — Sun, 26 Oct 2025 14:54:40 +0000

Running a highly available multi-cluster EKS architecture brings powerful benefits—zero downtime, disaster recovery, and global scalability. But it also multiplies your security challenges.

Securing a single EKS cluster is already complex. Add multiple clusters across regions, databases with sensitive data, and cross-cluster communication, and the attack surface grows significantly. One misconfigured security group or exposed secret can compromise your entire infrastructure.

This guide covers essential security considerations for multi-cluster architectures: network isolation, encryption, IAM management, secrets handling, and incident response. We'll focus on practical measures that protect your infrastructure without sacrificing performance or availability.

Let's build a secure, highly available system.

1. Network Security & Isolation

VPC Architecture

Separate VPCs per cluster or use shared VPC with isolated subnets
Private subnets for EKS nodes and databases (no direct internet access)
Public subnets only for load balancers and NAT gateways
Implement VPC peering or AWS Transit Gateway for inter-cluster communication
Use separate VPCs per environment (dev, staging, production)

Network Segmentation

Production VPC-1 (Region A)
- Public Subnets: ALB only
- Private Subnets: EKS Nodes
- Database Subnets: RDS/Aurora (isolated)
Production VPC-2 (Region B)
- Public Subnets: ALB only
- Private Subnets: EKS Nodes
- Database Subnets: RDS/Aurora (isolated)

Security Groups

Principle of least privilege - only allow necessary ports
Database security groups: Only allow traffic from EKS node security groups
EKS control plane: Restrict API access to specific CIDR ranges
No 0.0.0.0/0 rules except for outbound NAT traffic
Document and regularly audit security group rules

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-default
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-to-db
spec:
  podSelector:
    matchLabels:
      app: backend
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: database
    ports:
    - protocol: TCP
      port: 5432

2. Identity & Access Management (IAM)

Cluster Access Control

AWS IAM authentication for cluster access via aws-auth ConfigMap
Never use permanent credentials in pods or containers
Implement IAM Roles for Service Accounts (IRSA) for pod-level permissions
Use AWS SSO/IAM Identity Center for human access
Separate IAM roles for different teams/applications
Enable MFA for all human users

IRSA (IAM Roles for Service Accounts)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/app-role
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      serviceAccountName: app-sa
      containers:
      - name: app
        image: myapp:latest

Kubernetes RBAC

Role-Based Access Control (RBAC) for fine-grained permissions
Namespace isolation - separate namespaces per team/application
Principle of least privilege - minimal permissions needed
ClusterRoles for cluster-wide resources (use sparingly)
Roles for namespace-scoped resources
Regular RBAC audits

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: app-prod
rules:
- apiGroups: ["", "apps"]
  resources: ["pods", "deployments", "services"]
  verbs: ["get", "list", "watch"]
# Read-only access, no delete/update

Database Access

IAM Database Authentication for RDS/Aurora (where possible)
Avoid hardcoded credentials - use Secrets Manager or Parameter Store
Rotate credentials regularly (automated rotation)
Separate database users per application/service
Read-only replicas for non-critical workloads

3. Secrets Management

Never Store Secrets in Code or ConfigMaps
❌ Bad: Secrets in environment variables or ConfigMaps
✅ Good: External secrets management

AWS Secrets Manager / Parameter Store

Use External Secrets Operator or Secrets Store CSI Driver
Automatic rotation enabled
Encryption at rest with KMS
Audit access via CloudTrail

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
  target:
    name: db-secret
  data:
  - secretKey: password
    remoteRef:
      key: prod/db/password

Alternative: HashiCorp Vault

Centralized secrets management across clusters
Dynamic secrets generation
Lease-based credentials
Fine-grained access policies

4. Encryption

Data at Rest

EKS etcd encryption using AWS KMS
EBS volumes encrypted (gp3 with KMS)
RDS/Aurora encryption enabled with KMS
S3 encryption (SSE-S3 or SSE-KMS)
Use customer-managed KMS keys for compliance requirements
Separate KMS keys per environment/cluster

Data in Transit

TLS/SSL everywhere:
- ALB → Pods (via Ingress with TLS)
- Pod → Pod (service mesh or mTLS)
- Application → Database (SSL/TLS enforced)
Certificate management with AWS Certificate Manager or cert-manager
mTLS with service mesh (Istio, Linkerd, AWS App Mesh)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
spec:
  tls:
  - hosts:
    - app.example.com

5. Pod Security

Pod Security Standards

Enforce restricted pod security standards
No privileged containers unless absolutely necessary
Read-only root filesystem where possible
Non-root users for containers
Drop all capabilities and add only required ones

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL

Image Security

Scan images for vulnerabilities (Amazon ECR scanning, Trivy, Snyk)
Use minimal base images (distroless, Alpine)
Pin image versions - never use :latest
Sign and verify images (Sigstore/Cosign, Notary)
Private container registry (Amazon ECR with VPC endpoints)
Image pull secrets for private registries

Admission Controllers

OPA/Gatekeeper or Kyverno for policy enforcement
Enforce security policies:
No privileged pods
Required resource limits
Approved registries only
Required security contexts

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged
spec:
  validationFailureAction: enforce
  rules:
  - name: check-privileged
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged containers are not allowed"
      pattern:
        spec:
          containers:
          - securityContext:
              privileged: false

6. Multi-Cluster Security

Cluster Isolation

Separate clusters for different security zones (DMZ, internal, data)
Separate clusters per environment (never share prod and non-prod)
Separate AWS accounts per environment (AWS Organizations)
Service Control Policies (SCPs) to restrict actions at account level

Cross-Cluster Communication

Service mesh for secure cross-cluster communication (Istio multi-cluster)
VPC peering or Transit Gateway with strict security groups
mTLS for service-to-service authentication
API Gateway or Internal Load Balancer as entry points
Zero-trust networking - verify every request

DNS Security

Private Route53 hosted zones for internal services
DNSSEC where applicable
Avoid DNS-based service discovery across clusters (security risk)

7. Database Security

RDS/Aurora Security

Multi-AZ deployment for availability
Private subnets only - no public access
Encryption at rest (KMS) and in transit (SSL/TLS enforce)
Automated backups with encryption
Point-in-time recovery enabled
Enhanced monitoring enabled
Performance Insights with encryption

Connection Security

RDS Proxy for connection pooling and IAM authentication
SSL/TLS enforcement on database side
Certificate validation on client side
No hardcoded connection strings

# Database connection with SSL
DATABASE_URL: "postgresql://user@host:5432/db?sslmode=require"

Database Access Control

Separate database users per service
Minimal privileges (SELECT only for read-only services)
No superuser access from applications
Parameter groups to enforce security settings
Audit logging enabled (PostgreSQL pgaudit, MySQL audit log)

Database Activity Monitoring

AWS Database Activity Streams for real-time monitoring
Alert on suspicious queries or access patterns
Log all DDL and privilege changes

8. Logging & Monitoring

Comprehensive Logging

EKS Control Plane logs to CloudWatch (API, audit, authenticator)
Application logs via FluentBit/Fluentd to centralized location
Database logs (query logs, error logs, slow query logs)
VPC Flow Logs for network traffic analysis
CloudTrail for all API calls
Immutable logs - prevent tampering

Security Monitoring

Amazon GuardDuty - threat detection
AWS Security Hub - centralized security findings
Amazon Detective - security investigation
Falco - runtime security monitoring in Kubernetes
Prometheus + Grafana for metrics and alerting

Audit Logging

# Enable EKS audit logs
apiVersion: v1
kind: ConfigMap
metadata:
  name: audit-policy
data:
  policy.yaml: |
    apiVersion: audit.k8s.io/v1
    kind: Policy
    rules:
    - level: Metadata
      omitStages:
      - RequestReceived

Critical Alerts

Failed authentication attempts
Privileged container creation
Security group changes
Database connection failures
Unusual API calls
Resource exhaustion

9. Compliance & Governance

Compliance Frameworks

AWS Config - track configuration changes
AWS Audit Manager - compliance reporting
CIS Kubernetes Benchmark - security hardening
PCI-DSS, HIPAA, SOC 2 compliance where required
Regular penetration testing

Policy as Code

AWS Organizations with SCPs
CloudFormation/Terraform for infrastructure
GitOps for cluster configuration (ArgoCD/FluxCD)
OPA/Kyverno for admission control
Version control and peer review for all changes

Tagging Strategy

Mandatory tags: Environment, Owner, Project, CostCenter
Enforce tagging via AWS Config rules
Use tags for resource-level IAM policies
Cost allocation and chargeback

10. Disaster Recovery & Backup

Backup Strategy

Automated RDS snapshots (daily, 7-30 day retention)
Cross-region snapshot copies for DR
EBS snapshots for persistent volumes
etcd backups (Velero for cluster backups)
GitOps - cluster configuration in Git

Disaster Recovery

Multi-region setup for critical applications
RTO/RPO requirements documented
Failover procedures tested regularly
Regular DR drills (quarterly minimum)
Automated failover where possible (Route53 health checks)

11. Supply Chain Security

Container Supply Chain

Verify base images from trusted sources
SBOM (Software Bill of Materials) for dependencies
Vulnerability scanning in CI/CD pipeline
Sign images (Cosign/Notary)
Admission controller to verify signatures

Dependency Management

Dependabot or Renovate for automated updates
Regular security patching
Monitor CVEs for used software
Minimal dependencies principle

12. Incident Response

Preparation

Incident response plan documented
Runbooks for common scenarios
On-call rotation defined
Communication channels established
Post-mortem process defined

Detection & Response

Automated alerting for security events
Isolate compromised pods/nodes immediately
Forensics capability (preserve logs and state)
Contact AWS Support for suspected breaches
Notify stakeholders per incident severity

13. API Gateway & Service Mesh

API Security

AWS API Gateway or Kong/Envoy for API management
Rate limiting to prevent abuse
API keys or OAuth2 for authentication
WAF (Web Application Firewall) rules
DDoS protection via AWS Shield

Service Mesh Benefits

mTLS for all service-to-service communication
Zero-trust networking model
Fine-grained authorization policies
Observability and traffic monitoring
Circuit breaking and fault injection

14. Update & Patch Management

EKS Updates

Regular cluster updates (Kubernetes version support: ~14 months)
Test in non-prod first
Blue-green cluster upgrades for zero downtime
Node group rolling updates

Security Patching

Automated node updates (EKS managed node groups)
Bottlerocket OS for minimal attack surface
Container image updates (rebuild regularly)
Database patching during maintenance windows

Security Checklist

Private subnets for EKS nodes and databases
Security groups with least privilege
Network policies enforced
IRSA configured for all pods needing AWS access
No hardcoded credentials anywhere
Secrets Manager/Parameter Store with rotation
All encryption at rest enabled (etcd, EBS, RDS, S3)
TLS/SSL enforced everywhere
Pod security standards enforced (restricted)
Image scanning in CI/CD
Admission controllers (OPA/Kyverno) configured
GuardDuty and Security Hub enabled
Comprehensive logging to CloudWatch
CloudTrail enabled in all regions
VPC Flow Logs enabled
Regular backups with cross-region copies
Multi-factor authentication enforced
RBAC properly configured
Regular security audits scheduled
Incident response plan documented

Summary
Security in multi-cluster architectures requires a defense-in-depth approach:

Network isolation at every layer
Zero-trust model - verify everything
Encryption everywhere (at rest and in transit)
Least privilege access for humans and workloads
Continuous monitoring and alerting
Regular audits and compliance checks
Automation for consistency and reliability

Security is not a one-time setup but an ongoing process requiring continuous improvement and vigilance.

Migrating from ECS to EKS - Key Considerations

Nowsath — Mon, 06 Oct 2025 04:06:43 +0000

Thinking about moving your containers from Amazon ECS to Amazon EKS? You're not alone. Many teams are making this shift to leverage Kubernetes flexibility and industry standard tooling.

While both ECS and EKS run containers on AWS, the migration isn't straightforward. ECS task definitions need to become Kubernetes manifests. Your IAM roles need restructuring. Your deployment pipelines need updates. And your team needs to learn Kubernetes.

This guide covers everything you need to consider: from converting your applications and configuring networking, to setting up monitoring and training your team. We'll focus on practical considerations that will help you plan and execute a smooth migration.

Whether you're moving one service or your entire platform, understanding these key areas will save you time and headaches.

Let's get started with the following considerations..

1. Architecture & Workload Analysis

Assess Current ECS Setup:

Task Definitions → Need conversion to Kubernetes Deployments/StatefulSets
Service types (Fargate vs EC2) → Node groups or Fargate on EKS
Task placement strategies → Pod affinity/anti-affinity rules
Service discovery (Cloud Map, ALB) → Kubernetes Services/Ingress
Auto-scaling policies → HPA (Horizontal Pod Autoscaler)

Application Compatibility:

Review applications for ECS-specific dependencies
Check for AWS SDK calls that use ECS metadata endpoints
Identify hardcoded ECS configurations in application code
Assess container health check configurations

2. Container & Image Management

Container Images:

ECS and EKS both use Docker/OCI images - no changes needed
Ensure images are in Amazon ECR (or accessible registry)
Review image tagging strategy
Verify image security scanning is enabled

Startup Commands & Environment:

ECS command and entryPoint → Kubernetes command and args
Environment variables transfer directly
Secret handling needs migration (see below)

3. Configuration Translation

ECS Task Definition → Kubernetes Manifests

ECS Task Definition:

{
  "family": "my-app",
  "cpu": "512",
  "memory": "1024",
  "containerDefinitions": [{
    "name": "app",
    "image": "my-app:latest",
    "portMappings": [{"containerPort": 8080}]
  }]
}

Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: my-app:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "500m"
            memory: "1Gi"

Resource Mapping:

ECS CPU units (1024 = 1 vCPU) → Kubernetes millicores (1000m = 1 CPU)
ECS memory (MB) → Kubernetes memory (Mi/Gi)
Set both requests and limits in K8s

4. Networking Considerations

Load Balancing:

ECS ALB integration → AWS Load Balancer Controller for EKS
ECS Service Discovery (Cloud Map) → Kubernetes Services (DNS-based)
Target Groups → Managed by ALB Ingress Controller

Service Mesh (Optional):

Consider AWS App Mesh (works with both) or Istio/Linkerd for EKS
Service-to-service communication patterns
Traffic management and observability

Network Modes:

ECS awsvpc mode → Similar to Kubernetes pod networking
Choose appropriate CNI plugin (AWS VPC CNI, Calico, Cilium)
Plan IP address space (pods consume VPC IPs with AWS VPC CNI)

Security Groups:

ECS task-level security groups → SecurityGroupPolicy CRD in EKS
Or use Network Policies for pod-to-pod restrictions
Maintain database and service security group rules

5. IAM & Authentication

Task Roles → Service Accounts:

ECS Task Roles → IAM Roles for Service Accounts (IRSA)
Create OIDC provider for EKS cluster
Map IAM roles to Kubernetes service accounts
No more EC2 instance roles for pod permissions

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/my-app-role

Access Control:

ECS permissions → RBAC (Role-Based Access Control) in Kubernetes
AWS IAM → aws-auth ConfigMap for cluster access
Integrate with AWS SSO or IAM Identity Center

6. Secrets & Configuration Management

Secrets:
ECS Secrets (from Secrets Manager/SSM) → Multiple options:

AWS Secrets Manager/Parameter Store with External Secrets Operator
Kubernetes Secrets (less secure, stored in etcd)
Sealed Secrets or SOPS for GitOps

Configuration:

ECS environment variables → ConfigMaps and Secrets
ECS Systems Manager parameters → External Secrets Operator
Consider Helm for templating and configuration management

7. Storage & Persistence

Volumes:

ECS EFS volumes → EFS CSI Driver for EKS
ECS bind mounts → Kubernetes HostPath (not recommended) or emptyDir
ECS Docker volumes → Persistent Volume Claims (PVCs) with EBS CSI Driver

StatefulSets:

Applications requiring persistent storage need StatefulSets not Deployments
Define StorageClass and PersistentVolumeClaims

volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes: ["ReadWriteOnce"]
    storageClassName: gp3
    resources:
      requests:
        storage: 20Gi

8. Logging & Monitoring

Logging:

ECS CloudWatch Logs → Multiple options:

FluentBit/Fluentd DaemonSet → CloudWatch/S3/Elasticsearch
CloudWatch Container Insights for EKS
Maintain same log groups or migrate to new structure

Monitoring:

ECS metrics → Prometheus + Grafana ecosystem
CloudWatch Container Insights for both
Application Performance Monitoring (APM) tools (Datadog, New Relic, etc.)
Migrate existing dashboards and alerts

Tracing:

Ensure distributed tracing (X-Ray, Jaeger) continues to work
Update SDK configurations if needed

9. CI/CD Pipeline Changes

Deployment Pipeline:

Update pipelines to use kubectl, Helm, or GitOps (ArgoCD/FluxCD)
Replace ECS task definition updates with kubectl apply or Helm upgrades
Image building remains the same (push to ECR)

Deployment Strategies:

ECS rolling updates → Kubernetes rolling updates (more control)
Configure RollingUpdate strategy with maxSurge and maxUnavailable
Blue/green deployments using Argo Rollouts or Flagger

Testing:

Update integration tests for Kubernetes environment
Test in staging EKS cluster first
Validate health checks and readiness probes

10. Cost Considerations

Cost Changes:

EKS Control Plane: $0.10/hour per cluster (~$73/month)
Worker nodes: Similar to ECS EC2 instances
Fargate on EKS: Available but pricing differs from ECS Fargate
Potential for better resource utilization with Kubernetes bin-packing

Optimization:

Use Spot instances for cost savings
Implement Cluster Autoscaler or Karpenter
Right-size pods with appropriate resource requests/limits
Consider Savings Plans or Reserved Instances

11. Migration Strategy

Phased Approach (Recommended):

Phase 1: Preparation:

Set up EKS cluster in same VPC
Install necessary controllers (ALB, EBS CSI, etc.)
Convert task definitions to Kubernetes manifests
Set up monitoring and logging

Phase 2: Pilot Migration:

Migrate non-critical, stateless services first
Run in parallel with ECS (blue/green at service level)
Validate functionality and performance
Gather team feedback

Phase 3: Gradual Migration:

Service-by-service migration
Update DNS/load balancer routing gradually
Monitor each migration closely
Keep ECS as fallback initially

Phase 4: Complete Migration:

Migrate remaining services
Decommission ECS cluster
Update documentation and runbooks

Big Bang vs Strangler Pattern:

Big Bang: All at once (high risk, faster)
Strangler Pattern: Gradual service-by-service (lower risk, recommended)

12. Operational Changes

Team Training:

Train teams on Kubernetes concepts (pods, services, namespaces, etc.)
Learn kubectl commands and workflows
Understand YAML manifests and Helm charts
Practice troubleshooting techniques

New Tools & Skills:

kubectl (CLI tool)
Helm (package manager)
Kustomize (configuration management)
GitOps tools (ArgoCD, FluxCD)
Kubernetes dashboard or Lens (GUI tools)

Runbooks & Documentation:

Update incident response procedures
Document new deployment processes
Create troubleshooting guides for Kubernetes
Update disaster recovery plans

13. Testing & Validation

Pre-Migration Testing:

Load testing in EKS environment
Failover testing (pod crashes, node failures)
Database connection testing
Performance benchmarking vs ECS

Post-Migration Validation:

Monitor error rates and latency
Verify all integrations working
Check cost metrics
Validate backup/restore procedures
Test rollback procedures

14. Common Pitfalls to Avoid

⚠️ Resource limits: Not setting proper requests/limits causes instability
⚠️ Health checks: Incorrect probes cause pod restarts
⚠️ Secrets exposure: Storing secrets in ConfigMaps instead of Secrets
⚠️ Network policies: Forgetting to configure proper pod communication
⚠️ Storage: Not planning for persistent storage needs
⚠️ IAM roles: Not properly configuring IRSA
⚠️ Monitoring gaps: Losing observability during migration
⚠️ DNS caching: Application DNS caching issues with service discovery

15. Migration Checklist

EKS cluster setup with multi-AZ configuration
Install AWS Load Balancer Controller
Install EBS and EFS CSI drivers
Configure IAM OIDC provider
Set up IRSA for applications
Convert all task definitions to K8s manifests
Migrate secrets to Secrets Manager with External Secrets Operator
Configure monitoring and logging
Set up CI/CD pipelines for K8s
Update security groups and network policies
Test in staging environment
Create rollback plan
Train team on Kubernetes operations
Update documentation
Plan maintenance window for cutover

Summary
The migration from ECS to EKS is significant but manageable with proper planning. Key success factors:

Gradual migration (service by service)
Thorough testing at each stage
Team training on Kubernetes
Robust monitoring throughout the process
Clear rollback plans

Conclusion

Migrating from ECS to EKS is a significant undertaking, but with proper planning and a phased approach, it's absolutely achievable. The key is to start small, test thoroughly, and migrate service by service rather than attempting everything at once.

Remember that this migration isn't just about changing technology. It's about adopting a new way of thinking about container orchestration. Invest time in training your team, building robust monitoring, and documenting your processes. The flexibility and ecosystem that Kubernetes provides will be worth the effort.

Take your time, plan carefully, and don't hesitate to run both platforms in parallel during the transition. Your future self will thank you for the careful approach.

Happy migrating! 🤝😊

Build AWS Cloud Services Hangman Game with Amazon Q CLI

Nowsath — Tue, 03 Jun 2025 18:09:13 +0000

Are you looking for a fun way to learn AWS service names while enjoying a classic game? In this blog post, I'll walk you through an AWS-themed Hangman game I
built using Python and Pygame. This educational game helps players become familiar with AWS service names across different categories - perfect for AWS
certification candidates or anyone interested in cloud computing.

Game Overview

The AWS Cloud Services Hangman game challenges players to guess AWS service names one letter at a time. With 8 different AWS service categories and over 80
services to guess, it's both entertaining and educational.

Key Features:

• Multiple AWS service categories (Compute, Storage, Database, etc.)
• Interactive letter selection
• Visual hangman drawing that builds with each wrong guess
• Score tracking
• Custom AWS-themed background
• Clean and intuitive user interface

How the Game Works

When you start the game, you're presented with a menu screen featuring the AWS Cloud Services title. After clicking "Play Game," you select from various AWS
categories like Compute, Storage, or Database. The game then randomly selects an AWS service from that category for you to guess.

You have six attempts to guess the service name correctly. With each incorrect guess, another part of the hangman is drawn. Guess correctly, and you'll earn
a point. After completing 10 rounds, you'll see your final score.

Code Breakdown

Let's look at the key components of the code to understand how the game works:

1. Game Structure

The game is built using object-oriented programming with two main classes:
• Button: Handles all interactive buttons in the game
• Hangman: The main game class that manages game states and logic

class Button:
    def __init__(self, x, y, width, height, text, color, hover_color, text_color=BLACK, font=font):
        # Button initialization code

    def draw(self, surface):
        # Draw the button on the screen

    def check_hover(self, pos):
        # Check if mouse is hovering over button

    def is_clicked(self, pos, event):
        # Check if button is clicked

2. AWS Service Categories

The game uses a dictionary structure to organize AWS services by category:

word_categories = {
    "AWS Compute": ["EC2", "LAMBDA", "FARGATE", "LIGHTSAIL", "BEANSTALK", ...],
    "AWS Storage": ["S3", "EBS", "EFS", "FSX", "GLACIER", ...],
    "AWS Database": ["RDS", "DYNAMODB", "AURORA", "REDSHIFT", ...],
    # More categories...
}

This makes it easy to add new services or entire categories through the custom_words.py file.

3. Game States

The game uses a state machine pattern to manage different screens:

def run(self):
    # Game loop
    while running:
        # Handle events based on game state
        if self.game_state == "menu":
            # Menu screen logic
        elif self.game_state == "category_select":
            # Category selection logic
        elif self.game_state == "playing":
            # Gameplay logic
        elif self.game_state == "game_over":
            # Game over screen logic

        # Draw the appropriate screen
        if self.game_state == "menu":
            self.draw_menu()
        elif self.game_state == "category_select":
            self.draw_category_select()
        # And so on...

This approach makes the code modular and easier to maintain.

4. Background Image Handling

The game loads a background image from the local images folder:

# Load background image from local file
images_dir = os.path.join(os.path.dirname(__file__), "images")
bg_path = os.path.join(images_dir, "aws_bg.jpg")
if os.path.exists(bg_path):
    background_images["main"] = pygame.image.load(bg_path)
    background_images["main"] = pygame.transform.scale(background_images["main"],(SCREEN_WIDTH, SCREEN_HEIGHT))

A semi-transparent overlay is added to ensure text remains readable:

# Create a semi-transparent overlay for better text readability
overlay = pygame.Surface((SCREEN_WIDTH, SCREEN_HEIGHT), pygame.SRCALPHA)
overlay.fill((255, 255, 255, 180))  # White with 70% opacity

5. Drawing the Hangman

The hangman drawing is created step by step as the player makes incorrect guesses:

def draw_hangman(self):
    # Base
    pygame.draw.line(screen, BLACK, (150, 350), (250, 350), 5)

    # Pole
    if self.wrong_guesses >= 1:
        pygame.draw.line(screen, BLACK, (200, 350), (200, 100), 5)

    # Top beam
    if self.wrong_guesses >= 2:
        pygame.draw.line(screen, BLACK, (200, 100), (300, 100), 5)

    # And so on for each part of the hangman...

Educational Value

Beyond being a fun game, this project serves several educational purposes:

AWS Service Familiarity: Players naturally memorize AWS service names through repeated gameplay
Python Programming: The code demonstrates object-oriented programming, state machines, and event handling
Pygame Framework: Shows how to build interactive games with Pygame
UI/UX Design: Implements a clean, intuitive interface with proper feedback

Customization Options

The game is designed to be easily customizable:

• Add New AWS Services: Edit the custom_words.py file to add more services.
• Change Background: Replace the aws_bg.jpg file in the images folder.
• Add Sound Effects: Place MP3 files in the sounds folder for audio feedback.

Conclusion

Building this AWS Cloud Services Hangman game was both fun and educational. It demonstrates how gaming mechanics can be used to make learning technical content more engaging. The modular code structure makes it easy to extend with new features or customize for different learning objectives.

What's truly remarkable is how Amazon Q made the development process incredibly streamlined. As an AI assistant, Amazon Q helped me rapidly prototype the game, debug issues, and implement new features with minimal effort. When I encountered challenges like adding background images or fixing button spacing, Amazon Q provided immediate solutions with clear explanations.

The collaborative development experience with Amazon Q transformed what could have been a complex coding project into an accessible and enjoyable process. Even developers with limited game development experience can leverage Amazon Q to build educational games like this one, receiving guidance on everything from Pygame fundamentals to AWS-specific implementations.

Whether you're studying for AWS certifications or just want to become more familiar with cloud services, this game provides an entertaining way to reinforce your knowledge. And with Amazon Q as your coding partner, creating your own educational games becomes an achievable goal for developers at any skill level.

The complete source code is available on GitHub, along with installation instructions and more detailed documentation. Happy coding and happy AWS learning!

Securely access Amazon EKS with GitHub Actions and OpenID Connect

Nowsath — Tue, 14 Jan 2025 10:07:03 +0000

Modern DevOps practices often involve leveraging GitHub Actions for CI/CD workflows and Amazon Elastic Kubernetes Service (EKS) for container orchestration. To minimize the reliance on static credentials and improve security, integrating GitHub Actions with AWS via OpenID Connect (OIDC) provides a robust solution.

This guide will walk you through securely accessing an Amazon EKS cluster from GitHub Actions using OIDC.

Why Use OpenID Connect?

Traditionally, GitHub Actions access AWS resources through static credentials stored as secrets. While effective, this method carries risks if credentials are accidentally exposed. OIDC eliminates the need for static credentials by leveraging short-lived, dynamically generated tokens. This approach improves security and simplifies credential management.

Prerequisites

An AWS account with an EKS cluster set up.
A GitHub repository configured for your project.
The AWS CLI installed locally for initial setup with proper permissions.

Step 1: Enable OIDC in Your AWS Account

1. Add the GitHub OIDC Identity Provider: Run the following AWS CLI command to establish a trust relationship with GitHub’s OIDC provider:

aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list $(openssl s_client -servername token.actions.githubusercontent.com -connect token.actions.githubusercontent.com:443 2>/dev/null | openssl x509 -fingerprint -noout -sha1 | awk -F= '{print $2}' | tr -d ':')

2. Verify the OIDC Provider: Check that the OIDC provider was successfully created:

aws iam list-open-id-connect-providers

Step 2: Create an IAM Role for GitHub Actions

1. Define the Trust Policy: Create a JSON file named trust-policy.json with the following content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:<OWNER>/<REPO>:ref:refs/heads/<BRANCH>"
        }
      }
    }
  ]
}

Replace <AWS_ACCOUNT_ID>,<OWNER>,<REPO> and <BRANCH> with your AWS account ID, GitHub repository owner, repository name, and branch name.

2. Create the Role: Use the AWS CLI to create the IAM role:

aws iam create-role --role-name GitHubActionsEKSRole --assume-role-policy-document file://trust-policy.json

3. Attach a Policy for EKS Access: Attach the necessary permissions to the role. For example, to allow access to EKS, attach the AmazonEKSClusterPolicy and AmazonEKSWorkerNodePolicy

aws iam attach-role-policy --role-name GitHubActionsEKSRole --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
aws iam attach-role-policy --role-name GitHubActionsEKSRole --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy

Step 3: Associate access entry

1. Add your role to the cluster: Update the cluster access entry to attach the GitHubActionsEKSRole role

aws eks create-access-entry --cluster-name <EKS_CLUSTER_NAME> --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:role/GitHubActionsEKSRole --type STANDARD

Replace <AWS_ACCOUNT_ID>,<EKS_CLUSTER_NAME> with your AWS account ID and cluster name.

For additional configuration options: click here

2. Assign policies to the role: Add policies to the GitHubActionsEKSRole based on your requirements.

aws eks associate-access-policy --cluster-name <EKS_CLUSTER_NAME> --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:role/GitHubActionsEKSRole \
  --access-scope type=cluster --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy

Replace <AWS_ACCOUNT_ID>,<EKS_CLUSTER_NAME> with your AWS account ID and cluster name.

For additional configuration options: click here

Step 4: Update GitHub Actions Workflow

1. Configure Permissions in Workflow File: Update your GitHub Actions workflow YAML file to use OIDC authentication. Below is an example:

name: Deploy to EKS

on:
  push:
    branches:
      - main

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v3
        with:
          role-to-assume: arn:aws:iam::<AWS_ACCOUNT_ID>:role/GitHubActionsEKSRole
          aws-region: <AWS_REGION>

      - name: Deploy to EKS
        run: |
          aws eks update-kubeconfig --name <EKS_CLUSTER_NAME> --region <AWS_REGION>
          kubectl apply -f deployment.yaml

Replace <AWS_ACCOUNT_ID>,<AWS_REGION>, and <EKS_CLUSTER_NAME> with your AWS account ID, region, and EKS cluster name. Also you can retrieve those values from github secrets.

Step 5: Test the Workflow

Push the updated workflow file to your repository and monitor the GitHub Actions workflow execution. Ensure that the deployment step successfully accesses the EKS cluster without errors.

Step 6: Secure Your Setup

Restrict Role Permissions: Minimize the permissions associated with the IAM role to adhere to the principle of least privilege.

Audit Usage: Regularly audit IAM roles and policies to ensure compliance and security.

Conclusion

By integrating GitHub Actions with Amazon EKS using OpenID Connect, you eliminate the need for static credentials, enhancing the security and simplicity of your CI/CD workflows. This setup aligns with modern DevOps best practices, providing a scalable and secure way to manage access to AWS resources.

Migrate a hosted zone to a different AWS account in few seconds!!

Nowsath — Tue, 26 Nov 2024 12:37:15 +0000

Migrating a hosted zone from one AWS account to another involves creating a new hosted zone in the target account, replicating the DNS records, and updating the domain's nameservers. Here's a step-by-step guide for manual and automated steps.

-- Manual Steps --

In this process, will guide you through migrating a hosted zone using the AWS CLI.

1. Export DNS Records from the Source Account

i. Install AWS CLI if not already installed.
ii. Use the following command to export the DNS records from the hosted zone in the source account:

aws route53 list-resource-record-sets --hosted-zone-id <source-hosted-zone-id> > dns-records.json

iii. Save the output file (dns-records.json), which contains all DNS records.

2. Create a New Hosted Zone in the Target Account

Log in to the target AWS account.
Navigate to Route 53 and create a new hosted zone with the same domain name.
Note the new hosted zone ID and nameservers assigned to the zone.

3. Import DNS Records to the Target Account

i. Use the exported dns-records.json to replicate the records.
ii. Transform the JSON file to match the change-resource-record-sets API format if needed. An example format looks like this:

{
    "Changes": [
        {
            "Action": "CREATE",
            "ResourceRecordSet": {
                "Name": "example.com.",
                "Type": "A",
                "TTL": 300,
                "ResourceRecords": [
                    {
                        "Value": "192.0.2.1"
                    }
                ]
            }
        }
    ]
}

iii. Import the records to the new hosted zone:

aws route53 change-resource-record-sets --hosted-zone-id <new-hosted-zone-id> --change-batch file://dns-records.json

4. Update the Domain's Nameservers

i. Go to your domain registrar (e.g., AWS Route 53, GoDaddy, Namecheap).
ii. Replace the nameservers with the ones provided in the new hosted zone.
iii. Wait for the DNS propagation, which can take up to 48 hours.

5. Verify the Migration

i. Use tools like DNS Checker to ensure the records are correctly propagating.
Confirm that the DNS records are functional and resolving to the expected values.

Tips

Avoid downtime: Keep both hosted zones active until propagation is complete.
Delegate permissions: If you need cross-account access, consider using AWS Resource Access Manager (RAM) or an IAM role for temporary access.
Automate the process: Use tools like Terraform or Route 53's APIs for larger migrations.

-- Automated Steps --

Here’s a Python script using boto3 (AWS SDK for Python) to automate the transformation and migration of DNS records between AWS accounts. This script will:

Export DNS records from the source account.
Transform them into the format required for importing.
Import the records into the target account.

Prerequisites

i. Install the required libraries:

pip3 install boto3

ii. Set up AWS CLI profiles for both accounts:

Source account: aws configure --profile source_account
Target account: aws configure --profile target_account

iii. Save the python script as migrate_dns.py

import boto3
import json

# Constants
SOURCE_PROFILE = "source_account"
TARGET_PROFILE = "target_account"
SOURCE_HOSTED_ZONE_ID = "source-hosted-zone-id"
TARGET_HOSTED_ZONE_ID = "target-hosted-zone-id"

def export_dns_records():
    """Export DNS records from the source hosted zone."""
    session = boto3.Session(profile_name=SOURCE_PROFILE)
    client = session.client("route53")
    response = client.list_resource_record_sets(HostedZoneId=SOURCE_HOSTED_ZONE_ID)

    # Save records to a file
    with open("dns_records.json", "w") as f:
        json.dump(response["ResourceRecordSets"], f, indent=4)
    print("DNS records exported to dns_records.json")

def transform_records():
    """Transform DNS records for importing to the target hosted zone."""
    with open("dns_records.json", "r") as f:
        records = json.load(f)

    changes = []
    for record in records:
        # Skip NS and SOA records
        if record["Type"] in ["NS", "SOA"]:
            continue

        change = {
            "Action": "CREATE",
            "ResourceRecordSet": {
                "Name": record["Name"],
                "Type": record["Type"],
                "TTL": record.get("TTL", 300),  # Default TTL
                "ResourceRecords": record.get("ResourceRecords", [])
            }
        }
        changes.append(change)

    # Save transformed records to a file
    with open("transformed_records.json", "w") as f:
        json.dump({"Changes": changes}, f, indent=4)
    print("Records transformed and saved to transformed_records.json")

def import_dns_records():
    """Import transformed DNS records into the target hosted zone."""
    session = boto3.Session(profile_name=TARGET_PROFILE)
    client = session.client("route53")

    with open("transformed_records.json", "r") as f:
        change_batch = json.load(f)

    response = client.change_resource_record_sets(
        HostedZoneId=TARGET_HOSTED_ZONE_ID,
        ChangeBatch=change_batch
    )
    print("DNS records imported successfully")
    print(f"Change info: {response['ChangeInfo']}")

if __name__ == "__main__":
    # Step 1: Export DNS records from source account
    export_dns_records()

    # Step 2: Transform records for the target account
    transform_records()

    # Step 3: Import DNS records into the target account
    import_dns_records()

How to Use

i. Replace source-hosted-zone-id and target-hosted-zone-id with the respective hosted zone IDs. Also replace profiles if you have created differently.
ii. Run the script:

python migrate_dns.py

Key Notes

SOA and NS records are skipped: These are automatically managed by AWS.
TTL fallback: If a record lacks a TTL, a default value of 300 seconds is applied.

Install IBM Db2 Community Edition on Amazon EC2 (Ubuntu)

Nowsath — Sun, 15 Sep 2024 20:09:30 +0000

In this tutorial, we'll guide you through the step-by-step process of installing IBM Db2 Community Edition on an Amazon EC2 Ubuntu instance. We'll cover everything from setting up the EC2 instance to configuring Db2 and creating your first database.

IBM Db2 Community Edition is a cloud-native database designed to deliver low-latency transactions and high resilience for both structured and unstructured data. As an entry-level edition of the Db2 data server, it's tailored for developers and partners to explore the capabilities of Db2 in a cost-effective manner.

Available for Linux, Windows, AIX, and as a Docker image, Db2 Community Edition offers a comprehensive feature set, including core Db2 functionalities. While it is limited to up to 4 cores and 16 GB RAM, it provides a solid foundation for building and testing applications.

Security is a top priority with Db2 Community Edition, featuring always-on security measures to protect your data. Additionally, the community-driven support model ensures that you have access to helpful resources and assistance from fellow developers.

Prerequisites

Setup an EC2 instance of type t2.medium
Ubuntu 22.04 LTS as AMI
30 GB of hard disk space
Open port 22 for SSH and 25000 for Db2
Create an IBM account for downloading Db2 community edition

Installation Steps

I. Login to your EC2 instance and verify the distribution version.

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

II. Copy the downloaded latest IBM Db2 Linux (x64) version to EC2 instance and extract it.

$ tar -xzvf v11.5.9_linuxx64_server_dec.tar.gz

$ ls
server_dec  v11.5.9_linuxx64_server_dec.tar.gz

Inside the server_dec directory, you can see following list of files.

~/server_dec$ ls
db2
db2_deinstall
db2_install  
db2checkCOL.tar.gz  
db2checkCOL_readme.txt  
db2ckupgrade  
db2ls  
db2prereqcheck  
db2setup  
installFixPack

III. From server_dec directory, Run db2prereqcheck command to check the prerequisite requirements for installing Db2.

~/server_dec$ sudo ./db2prereqcheck

==========================================================================

Sun Sep 15 16:37:59 2024
Checking prerequisites for DB2 installation. Version "11.5.9.0". Operating system "Linux" 

Validating "kernel level " ... 
   Required minimum operating system kernel level: "3.10.0". 
   Actual operating system kernel level: "6.5.0". 
   Requirement matched. 

Validating "Linux distribution " ... 
   Required minimum "UBUNTU" version: "16.04" 
   Actual version: "22.04" 
   Requirement matched. 

Validating "Bin user" ... 
   Requirement matched. 

Validating "C++ Library version " ... 
   Required minimum C++ library: "libstdc++.so.6" 
   Standard C++ library is located in the following directory: "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30". 
   Actual C++ library: "CXXABI_1.3.1" 
   Requirement matched. 


Validating "32 bit version of "libstdc++.so.6" " ... 
   Found the 64 bit "/lib/x86_64-linux-gnu/libstdc++.so.6" in the following directory "/lib/x86_64-linux-gnu". 
DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "libstdc++.so.6". 

Validating "libaio.so version " ... 
DBT3553I  The db2prereqcheck utility successfully loaded the libaio.so.1 file. 
   Requirement matched. 

Validating "libnuma.so version " ... 
DBT3610I  The db2prereqcheck utility successfully loaded the libnuma.so.1 file. 
   Requirement matched. 

Validating "/lib/i386-linux-gnu/libpam.so*" ... 
   DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "/lib/i386-linux-gnu/libpam.so*". 
   WARNING : Requirement not matched. 
Requirement not matched for DB2 database "Server" . Version: "11.5.9.0". 
Summary of prerequisites that are not met on the current system: 
   DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "/lib/i386-linux-gnu/libpam.so*". 


DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "libstdc++.so.6".

The output indicates that the requirement checks have failed, and we need to address these issues.

Enable 32-bit architecture on your instance and install the necessary packages.
Install 32-bit library file: "/lib/i386-linux-gnu/libpam.so*" packages.

$ sudo dpkg --add-architecture i386
$ sudo apt-get update -y
$ sudo apt-get install libstdc++6:i386 libpam0g:i386 -y

After installing the required packages, rerun the db2prereqcheck command to ensure all requirements are met.

$ sudo ./db2prereqcheck 

==========================================================================

Sun Sep 15 16:40:55 2024
Checking prerequisites for DB2 installation. Version "11.5.9.0". Operating system "Linux" 

Validating "kernel level " ... 
   Required minimum operating system kernel level: "3.10.0". 
   Actual operating system kernel level: "6.5.0". 
   Requirement matched. 

Validating "Linux distribution " ... 
   Required minimum "UBUNTU" version: "16.04" 
   Actual version: "22.04" 
   Requirement matched. 

Validating "Bin user" ... 
   Requirement matched. 

Validating "C++ Library version " ... 
   Required minimum C++ library: "libstdc++.so.6" 
   Standard C++ library is located in the following directory: "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30". 
   Actual C++ library: "CXXABI_1.3.1" 
   Requirement matched. 


Validating "32 bit version of "libstdc++.so.6" " ... 
   Found the 32 bit "/lib/i386-linux-gnu/libstdc++.so.6" in the following directory "/lib/i386-linux-gnu". 
   Requirement matched. 

Validating "libaio.so version " ... 
DBT3553I  The db2prereqcheck utility successfully loaded the libaio.so.1 file. 
   Requirement matched. 

Validating "libnuma.so version " ... 
DBT3610I  The db2prereqcheck utility successfully loaded the libnuma.so.1 file. 
   Requirement matched. 

Validating "/lib/i386-linux-gnu/libpam.so*" ... 
   Requirement matched. 
DBT3533I  The db2prereqcheck utility has confirmed that all installation prerequisites were met.

IV. Install Db2 using db2_install command as the root user and wait for the installation to complete.

$ sudo ./db2_install
Read the license agreement file in the db2/license directory.

***********************************************************
To accept those terms, enter "yes". Otherwise, enter "no" to cancel the install process. [yes/no]
yes


Default directory for installation of products - /opt/ibm/db2/V11.5

***********************************************************
Install into default directory (/opt/ibm/db2/V11.5) ? [yes/no] 
yes


Specify one of the following keywords to install DB2 products.

  SERVER 
  CONSV 
  CLIENT 
  RTCL 

Enter "help" to redisplay product names.

Enter "quit" to exit.

***********************************************************
SERVER
***********************************************************
Do you want to install the DB2 pureScale Feature? [yes/no] 
no
DB2 installation is being initialized.

Once the installation is completed, you will see the message below. You can then review the installation log file for post-installation steps.

The execution completed successfully.

For more information see the DB2 installation log at
"/tmp/db2_install.log.3355".

$ cat /tmp/db2_install.log.3355

Post-installation instructions 
-------------------------------

Required steps: 
Set up a DB2 instance to work with DB2. 

Optional steps: 
Notification SMTP server has not been specified. Notifications cannot be sent to contacts in your contact list until this is specified. For more information see the DB2 administration documentation. 

To validate your installation files, instance, and database functionality, run the Validation Tool, /opt/ibm/db2/V11.5/bin/db2val. For more information, see "db2val" in the DB2 Information Center. 

Refer to "What's new" https://www.ibm.com/docs/en/db2/11.5?topic=database-whats-new in the DB2 Information Center to learn about the new functions for DB2 V11.5. 

Open First Steps by running "db2fs" using a valid user ID such as the DB2 instance owner's ID. You will need to have DISPLAY set and a supported web browser in the path of this user ID. 

Verify that you have access to the DB2 Information Center based on the choices you made during this installation. If you performed a typical or a compact installation, verify that you can access the IBM Web site using the internet. If you performed a custom installation, verify that you can access the DB2 Information Center location specified during the installation. 

Ensure that you have the correct license entitlements for DB2 products and features installed on this machine. Each DB2 product or feature comes with a license certificate file (also referred to as a license key) that is distributed on an Activation CD, which also includes instructions for applying the license file. If you purchased a base DB2 product, as well as, separately priced features, you might need to install more than one license certificate. The Activation CD for your product or feature can be downloaded from Passport Advantage if it is not part of the physical media pack you received from IBM. For more information about licensing, search the Information Center (https://www.ibm.com/docs/en/db2/11.5) using terms such as "license compliance", "licensing" or "db2licm". 

To use your DB2 database product, you must have a valid license. For information about obtaining and applying DB2 license files, see http://www-01.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.qb.server.doc/doc/c0061199.html.

V. To validate our installation, execute the db2val command and review the log file to ensure everything is functioning correctly. If the validation completes without issues.

$ sudo /opt/ibm/db2/V11.5/bin/db2val
DBI1379I  The db2val command is running. This can take several minutes.

DBI1335I  Installation file validation for the DB2 copy installed at
      /opt/ibm/db2/V11.5 was successful.

DBI1343I  The db2val command completed successfully. For details, see
      the log file /tmp/db2val-240915_164309.log.

$ cat /tmp/db2val-240915_164309.log
Installation file validation for the DB2 copy installed at "/opt/ibm/db2/V11.5" starts.

Task 1: Validating Installation file sets.
Status 1 : Success 

Task 2: Validating embedded runtime path for DB2 executables and libraries.
Status 2 : Success 

Task 3: Validating the accessibility to the installation path.
Status 3 : Success 

Task 4: Validating the accessibility to the /etc/services file.
Status 4 : Success 

DBI1335I  Installation file validation for the DB2 copy installed at
      /opt/ibm/db2/V11.5 was successful.

Installation file validation for the DB2 copy installed at "/opt/ibm/db2/V11.5" ends.

DBI1343I  The db2val command completed successfully. For details, see
      the log file /tmp/db2val-240915_164309.log.

Post Installation Steps

I. Create groups and users for DB administrators and fenced users, then assign passwords to the users.

$ sudo groupadd -g 997 db2admin
$ sudo groupadd -g 998 db2fence


$ sudo useradd -u 1003 -g db2admin -m -d /home/db2adm1 db2adm1
$ sudo useradd -u 1004 -g db2fence -m -d /home/db2fen1 db2fen1

II. Create a Db2 instance using the db2icrt command and check the log file for details on database connectivity.

$ sudo /opt/ibm/db2/V11.5/instance/db2icrt -a server -u db2fen1 db2adm1
DBI1446I  The db2icrt command is running.


DB2 installation is being initialized.

 Total number of tasks to be performed: 4 
Total estimated time for all tasks to be performed: 309 second(s) 

Task #1 start
Description: Setting default global profile registry variables 
Estimated time 1 second(s) 
Task #1 end 

Task #2 start
Description: Initializing instance list 
Estimated time 5 second(s) 
Task #2 end 

Task #3 start
Description: Configuring DB2 instances 
Estimated time 300 second(s) 
Task #3 end 

Task #4 start
Description: Updating global profile registry 
Estimated time 3 second(s) 
Task #4 end 

The execution completed successfully.

For more information see the DB2 installation log at "/tmp/db2icrt.log.80519".
DBI1070I  Program db2icrt completed successfully.

$ cat /tmp/db2icrt.log.80519

Post-installation instructions 
-------------------------------

Required steps: 
You can connect to the DB2 instance "db2adm1" using the port number "25001". Record it for future reference.

III. Now start the database instance using the below commands.

$ sudo su - db2adm1


$ db2ls

Install Path                       Level   Fix Pack   Special Install Number   Install Date                  Installer UID 
---------------------------------------------------------------------------------------------------------------------
/opt/ibm/db2/V11.5               11.5.9.0        0                            Sun Sep 15 16:42:52 2024 UTC             0

$ . sqllib/userprofile

$ db2start 
09/15/2024 19:52:45     0   0   SQL1063N  DB2START processing was successful.
SQL1063N  DB2START processing was successful.

IV. Now the database instance is started and we can connect to the instance.

$ db2
(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 11.5.9.0

You can issue database manager commands and SQL statements from the command 
prompt. For example:
    db2 => connect to sample
    db2 => bind sample.bnd

For general help, type: ?.
For command help, type: ? command, where command can be
the first few keywords of a database manager command. For example:
 ? CATALOG DATABASE for help on the CATALOG DATABASE command
 ? CATALOG          for help on all of the CATALOG commands.

To exit db2 interactive mode, type QUIT at the command prompt. Outside 
interactive mode, all commands must be prefixed with 'db2'.
To list the current command option settings, type LIST COMMAND OPTIONS.

For more detailed help, refer to the Online Reference Manual.

db2 =>

V. Create a test database and connect to the database using the below commands.

db2 => create database sales
DB20000I  The CREATE DATABASE command completed successfully.

db2 => connect to sales

   Database Connection Information

 Database server        = DB2/LINUXX8664 11.5.9.0
 SQL authorization ID   = DB2ADM1
 Local database alias   = SALES

That's it! Now you're ready to explore and work with the IBM Db2 database. I hope you find this guide helpful and enjoyable!

Troubleshooting: GitHub Actions + Terraform + EKS + Helm

Nowsath — Sun, 01 Sep 2024 16:24:21 +0000

In this post, I've compiled a list of issues I encountered while setting up an EKS Fargate cluster using Terraform code and GitHub Actions, along with their corresponding solutions.

Error 1: Connect: connection refused for service account.



│ Error: Get "http://localhost/api/v1/namespaces/kube-system/serviceaccounts/aws-load-balancer-controller": dial tcp [::1]:80: connect: connection refused

│ 
│   with kubernetes_service_account.service-account,
│   on aws-alb-controller.tf line 50, in resource "kubernetes_service_account" "service-account":
│   50: resource "kubernetes_service_account" "service-account" {
│

Error 2: Connect: connection refused for namespace.



│ Error: Get "http://localhost/api/v1/namespaces/aws-observability": dial tcp [::1]:80: connect: connection refused
│ 
│   with kubernetes_namespace.aws_observability,
│   on namespaces.tf line 1, in resource "kubernetes_namespace" "aws_observability":
│    1: resource "kubernetes_namespace" "aws_observability" {

Error 3: Kubernetes cluster unreachable.



│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ 
│   with helm_release.alb-controller,
│   on aws-alb-controller.tf line 69, in resource "helm_release" "alb-controller":
│   69: resource "helm_release" "alb-controller" {

Solutions for above errors 1, 2, 3:
These issues often occur when you attempt to run Terraform commands for updates on an EKS cluster after its creation using the EKS module.

If you've used the Kubernetes provider as shown below, you're likely to encounter these types of errors.



provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

To resolve this, you should configure the Kubernetes provider as follows:



provider "kubernetes" {
  host = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

In this configuration, the Kubernetes provider's host and cluster_ca_certificate values are sourced directly from the outputs of the eks module.

Pre-computed Outputs: The module.eks.cluster_endpoint and module.eks.cluster_certificate_authority_data are outputs that are likely computed and stored when the EKS module is applied. This means that once the module is applied, these outputs remain constant unless the cluster configuration changes.
No Additional Data Source Calls: Since the endpoint and certificate authority data are provided by the module's outputs, Terraform doesn't need to make an additional API call to AWS to retrieve this information each time you run terraform plan or apply. This reduces the likelihood of timing issues or inconsistencies that might cause connection problems.

Error 4: Connect: Missing required argument.



│ Error: Missing required argument
│ 
│   with data.aws_eks_cluster_auth.cluster,
│   on aws-alb-controller.tf line 21, in data "aws_eks_cluster_auth" "cluster":
│   21:   name = module.eks.cluster_id
│ 
│ The argument "name" is required, but no definition was found.

Solutions:

This error suggests that the name argument is required for the aws_eks_cluster_auth data source, but it's not being provided correctly. Specifically, the module.eks.cluster_id is not returning a value, or the value it is returning is not recognized as the name of the EKS cluster.

i. Check the output of module.eks.cluster_id: Ensure that module.eks.cluster_id is correctly referencing the EKS cluster ID. It should be an output of the EKS module. Verify that the cluster_id output is correctly defined in your EKS module.



output "cluster_id" {
  value = module.eks.cluster_id
}

ii. Verify the EKS Module Version: Make sure you are using the correct version of the terraform-aws-modules/eks/aws module. If the module version is incorrect or outdated, it may not include the cluster_id output.

iii. Direct Reference: If the module output is correctly defined but still causing issues, try directly referencing the cluster name instead of the 'cluster_id'



data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
}

Error 5: Required plugins are not installed.




│ Error: Required plugins are not installed
│ 
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.terraform.io/hashicorp/null: there is no package for registry.terraform.io/hashicorp/null 3.2.2 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/time: there is no package for registry.terraform.io/hashicorp/time 0.11.2 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/tls: there is no package for registry.terraform.io/hashicorp/tls 4.0.5 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/aws: there is no package for registry.terraform.io/hashicorp/aws 5.53.0 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/cloudinit: there is no package for registry.terraform.io/hashicorp/cloudinit 2.3.4 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/helm: there is no package for registry.terraform.io/hashicorp/helm 2.14.0 cached in .terraform/providers
│   - registry.terraform.io/hashicorp/kubernetes: there is no package for registry.terraform.io/hashicorp/kubernetes 2.31.0 cached in .terraform/providers
│ 
│ Terraform uses external plugins to integrate with a variety of different
│ infrastructure services. To download the plugins required for this
│ configuration, run:
│   terraform init

Solutions:

The error you're seeing indicates that Terraform's provider plugins are not correctly installed or do not match the versions specified in your dependency lock file (.terraform.lock.hcl). This can happen if you haven't run terraform init after modifying your Terraform configuration or if the plugins have been removed from the cache.

Here’s how to resolve this issue:

i. Run terraform init: The error message itself suggests running terraform init. This command initializes the Terraform working directory by downloading the required provider plugins and setting up the necessary backend.



terraform init

This should download and install all the required plugins based on your configuration and lock file.

ii. Force Re-initialization: If running terraform init alone does not solve the problem, you can try forcing a reinitialization, which will re-download the providers and update the .terraform directory:



terraform init -upgrade

The -upgrade flag will ensure that Terraform checks for the latest versions of the required plugins and updates your local cache accordingly.

iii. Clear and Re-initialize the Terraform Directory: If the above steps still don’t resolve the issue, you may want to clear out the .terraform directory and reinitialize from scratch:



rm -rf .terraform
terraform init

This will delete the existing .terraform directory (where Terraform caches plugins and stores state-related files) and then reinitialize the working directory.

Error 6: Backend configuration changed.



│ Error: Backend configuration changed
│ 
│ A change in the backend configuration has been detected, which may require migrating existing state.
│ 
│ If you wish to attempt automatic migration of the state, use "terraform init -migrate-state".
│ If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".

Solutions:

The error message you're seeing indicates that Terraform has detected a change in your backend configuration. The backend is where Terraform stores the state file, which keeps track of your infrastructure resources. When the backend configuration changes, Terraform needs to decide how to handle the existing state.

You have two options depending on your situation:
i. Automatic State Migration:

When to Use: If you've changed the backend configuration (e.g., switched from local state to a remote backend like S3 or changed the backend's parameters), and you want Terraform to automatically migrate your existing state to the new backend.



terraform init -migrate-state

What It Does: This command will attempt to migrate your existing state file to the new backend. It's a safe option if you're confident that the backend change is intentional and you want to keep your infrastructure's state intact in the new location.

ii. Reconfigure Without Migrating State:

When to Use: If you made changes to the backend configuration but do not need to migrate the state (e.g., you're working in a different environment, or you've changed a configuration that doesn't affect the state).



terraform init -reconfigure

What It Does: This command reconfigures the backend without migrating the state. It is useful if you're setting up Terraform in a new environment or if the state migration is unnecessary or handled separately.

Error 7: Helm release "" was created but has a failed status.



│ Warning: Helm release "" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ 
│   with helm_release.alb-controller,
│   on aws-alb-controller.tf line 75, in resource "helm_release" "alb-controller":
│   75: resource "helm_release" "alb-controller" {

│ Warning: Helm release "" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│ 
│   with helm_release.alb-controller,
│   on aws-alb-controller.tf line 75, in resource "helm_release" "alb-controller":
│   75: resource "helm_release" "alb-controller" {


│ Error: context deadline exceeded
│ 
│   with helm_release.alb-controller,
│   on aws-alb-controller.tf line 75, in resource "helm_release" "alb-controller":
│   75: resource "helm_release" "alb-controller" {

Solutions:

The error you're encountering indicates that the Helm release for the ALB (Application Load Balancer) controller was created but ended up in a failed state. Additionally, the "context deadline exceeded" error suggests that the operation took too long to complete, possibly due to network issues, resource constraints, or misconfiguration.

To fix this, run 'terraform apply' again it will automatically replace the tainted helm_release.alb-controller as shown in the below image.

Error 8: No value for required variable.



│ Error: No value for required variable
│ 
│   on variables.tf line 85:
│   85: variable "route53_access_key" {
│ 
│ The root module input variable "route53_access_key" is not set, and has no
│ default value. Use a -var or -var-file command line argument to provide a
│ value for this variable.

Solutions:

The error message indicates that the Terraform configuration requires an input variable named route53_access_key, which has not been provided a value and doesn't have a default value set in your variables.tf file.

Error 9: InvalidArgument: The parameter Origin DomainName does not refer to a valid S3 bucket.



│ Error: updating CloudFront Distribution (E1B4377XXXX): operation error CloudFront: UpdateDistribution, https response error StatusCode: 400, RequestID: 0b25a733-e79-4xx7-bxx6-5exxx1cb405, InvalidArgument: The parameter Origin DomainName does not refer to a valid S3 bucket.
│ 
│   with aws_cloudfront_distribution.cms_cf,
│   on cloudfronts.tf line 1, in resource "aws_cloudfront_distribution" "cms_cf":
│    1: resource "aws_cloudfront_distribution" "cms_cf" {

Solutions:

The error message indicates that the Origin DomainName specified in your CloudFront distribution configuration does not point to a valid S3 bucket. This commonly happens when:

i. Incorrect S3 bucket name: Ensure that the S3 bucket name you are using in your CloudFront distribution is correct and exists.

ii. S3 bucket URL format: The Origin DomainName for an S3 bucket should follow this pattern:

For a S3 bucket: bucket-name.s3.amazonaws.com
For a regional S3 bucket: bucket-name.s3.<region>.amazonaws.com

iii. Bucket permissions: Ensure the S3 bucket has the correct permissions to allow access from CloudFront. The bucket policy should allow CloudFront access.

You can check the origin block in your CloudFront resource, which should look like this:



resource "aws_cloudfront_distribution" "cms_cf" {
  origin {
    domain_name = "your-bucket-name.s3.amazonaws.com"
    origin_id   = "S3-your-bucket-name"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.example.cloudfront_access_identity_path
    }
  }

  // Other CloudFront configuration
}

Error 10: Reference to undeclared input variable.



│ Error: Reference to undeclared input variable
│ 
│   on eks.tf line 65 in module "eks":
│   65:       namespace = "${var.namespace[0]}"
│ 
│ An input variable with the name "namespace" has not been declared. | Did you mean "namespaces?

Solutions:

As this error message indicates that in your eks.tf file, you are trying to reference the variable namespace, but it has not been declared.

Check if the intended variable is namespace or namespaces, and adjust the code and declaration accordingly. Also ensure that the variable is declared in your variables.tf or in the same file where you're using it.

Thanks for reading! I'll update this post regularly. Please share any other issues in the comments. 🤝😊

Differences between primary, core and task nodes in Amazon EMR cluster

Nowsath — Sun, 09 Jun 2024 10:43:42 +0000

The key differences between primary, core, and task nodes in an Amazon EMR cluster are:

Primary Node (also known as Master Node):

The primary node is responsible for coordinating the cluster and managing the execution of jobs.
It runs the main Hadoop services, such as the JobTracker, NameNode, and ResourceManager.
There is only one primary node in an EMR cluster.
The primary node cannot be terminated during the lifetime of the cluster, as it is essential for the cluster's operation.

Core Nodes:

Core nodes host the Hadoop Distributed File System (HDFS) and run the DataNode and TaskTracker services.
They are responsible for storing and processing data in the cluster.
Core nodes cannot be removed from the cluster without risking data loss, as they contain the persistent data in HDFS.
You should reserve core nodes for the capacity that is required until your cluster completes.

Task Nodes:

Task nodes are used for running tasks and do not host HDFS. They can be added or removed from the cluster as needed, without the risk of data loss.
Task nodes are ideal for handling temporary or burst workloads, as you can launch task instance fleets on Spot Instances to increase capacity while minimizing costs.
The cluster will never scale below the minimum constraints set in the managed scaling policy.

Here's a table summarizing the key differences:

More details regarding,

Selecting and deploying an Amazon EMR cluster: click here
Estimating Amazon EMR cluster capacity: click here

EKS cluster upgrade fail with Kubelet version of Fargate pods must be updated to match cluster version!!

Nowsath — Sat, 30 Dec 2023 17:04:24 +0000

You might encounter the following error message within the AWS EKS console when attempting to upgrade the Kubernetes version of your cluster.

Kubelet version of Fargate pods must be updated to match cluster version 1.25 before updating cluster version; Please recycle all offending pod replicas.

For instance, if you are in the process of incrementally upgrading the Kubernetes version from 1.24 to 1.27.

The upgrade from version 1.24 to 1.25 will proceed without encountering errors. However, please note that it won’t automatically update the kubelet version of the Fargate nodes/Pods until a restart is performed.

But when you check Fargate nodes versions,

Or when you check from terminal,



# To list all nodes
kubectl get nodes

Solutions..

If the Fargate pods’ versions are not compatible with the upgraded EKS cluster, you might need to recreate the pods using a compatible version.

You can do this by updating the deployment or stateful set YAML files with the correct version and applying the changes to the cluster. Kubernetes will automatically create new pods with the updated version.
You can get all deployment, statefulset from all namespaces and restart it as shown below.



# To list all deployments
kubectl get deployment -A

# To list all deployments
kubectl get statefulset -A

# To restart deployments
kubectl rollout restart deployment coredns -n kube-system
kubectl rollout restart deployment metrics-server -n kube-system
kubectl rollout restart deployment aws-load-balancer-controller -n kube-system
kubectl rollout restart deployment ebs-csi-controller -n kube-system

# To restart statefulset
kubectl rollout restart statefulset adot-collector -n fargate-container-insights

Please note that this solution will work exclusively with Fargate nodes. If you are using other cluster modes, please refer to the links provided at the end of this article for alternative solutions.

Also don’t forget to update kubectl as well.



# To check kubectl version
kubectl version --short --client

Link to update kubectl:
Installing or updating kubectl

For more details about Amazon EKS Kubernetes versions upgrade:
Amazon EKS Kubernetes versions
Updating an Amazon EKS cluster Kubernetes version

I’ll always strive to provide simple explanations. If you found this article useful, follow me for more similar articles in the future. Your support keeps me motivated to create content like this. 🤝😊

Setup Prometheus and Grafana with existing EKS Fargate cluster - Monitoring

Nowsath — Fri, 29 Dec 2023 13:30:43 +0000

In this article, I will delineate the fundamental steps for configuring Prometheus and Grafana within the existing EKS Fargate cluster, along with the establishment of custom metrics. These measures are commonly utilized for monitoring and alerting purposes.

Steps to follow.

Configure Node Groups
Install AWS EBS CSI driver
Install Prometheus
Install Grafana

1. Configure Node Groups

Given the absence of any pre-existing node groups, let's proceed to create a new one.

i. Create an IAM Role for EC2 worker nodes

Go to the AWS IAM console, create a role 'AmazonEKSWorkerNodeRole' with following 3 AWS managed policies.

AmazonEC2ContainerRegistryReadOnly
AmazonEKS_CNI_Policy
AmazonEKSWorkerNodePolicy

ii. Create Node groups

Navigate to the AWS EKS console, add node group to the cluster. It is imperative to utilize EC2 instances for Prometheus and Grafana, as both applications require volumes to be mounted on them.

When configuring the node group for the cluster, take the following into consideration,

Node IAM role: Select created role in the previous step (AmazonEKSWorkerNodeRole)
Instance type: Select based on your requirements (t3.small in my case)
Subnets: The private subnets within the VPC where the EKS cluster is located. If you want to enable remote access to node then you need to use public subnets.

In my configuration, I opted for a t3.small instance with a desired size of 1. The setup worked without any issues.

To verify the proper functioning of the EC2 worker nodes, execute the following command. The output should indicate that a pod is currently running.



$ k get po -l k8s-app=aws-node -n kube-system
NAME             READY   STATUS    RESTARTS   AGE
aws-node-hbvz2   1/1     Running   0          58m

2. Install AWS EBS CSI driver

The Amazon EBS CSI driver manages the lifecycle of Amazon EBS volumes as storage for the Kubernetes Volumes that you create.

The Amazon EBS CSI driver makes Amazon EBS volumes for these types of Kubernetes volumes: generic ephemeral volumes and persistent volumes.

Prometheus and Grafana require persistent storage, commonly referred to as PV (Persistent Volume) in Kubernetes terminology, to be attached to them.

i. Create AWS EBS CSI driver IAM role and associate to service account

Create a service account named 'ebs-csi-controller-sa' and associate it with AWS managed IAM policies. This service account will be utilized during the installation of the AWS EBS CSI driver.

Replace 'my-cluster' with the name of your cluster.



eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --role-name AmazonEKS_EBS_CSI_DriverRole \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --override-existing-serviceaccounts --approve

If you don't specify the role name it will assign a name automatically.

ii. Add Helm repositories

We will use Helm to install the components required to run Prometheus and Grafana.



helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

ii. Installing aws-ebs-csi-driver

Following the addition of the new Helm repository, proceed to install the AWS EBS CSI driver using the below helm command.

Replace the region 'eu-north-1' with your cluster region.



helm upgrade --install aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.region=eu-north-1 \
  --set controller.serviceAccount.create=false \
  --set controller.serviceAccount.name=ebs-csi-controller-sa \
  aws-ebs-csi-driver/aws-ebs-csi-driver

3. Install Prometheus

For persistent storage of scraped metrics and configurations, Prometheus leverages two EBS volumes: one dedicated to the prometheus-server pod and another for the prometheus-alertmanager pod.

i. Create a namespace for Prometheus

Create a namespace called 'prometheus'.
kubectl create namespace prometheus

ii. Set availability Zone and create storage class

There are two options for storage class:

Use the default storage class (proceed to step iii).
Create a storage class in your worker node's Availability Zone (AZ).

Get the Availability Zone of one of the worker nodes:



EBS_AZ=$(kubectl get nodes \
  -o=jsonpath="{.items[0].metadata.labels['topology\.kubernetes\.io\/zone']}")

Create a storage class:



echo "
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: prometheus
  namespace: prometheus
provisioner: ebs.csi.aws.com
parameters:
  type: gp2
reclaimPolicy: Retain
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - $EBS_AZ
" | kubectl apply -f -

iii. Installing Prometheus

First download the Helm values for Prometheus file:



wget https://github.com/aws-samples/containers-blog-maelstrom/raw/main/fargate-monitoring/prometheus_values.yml

Moreover, if you wish to configure custom metrics endpoints, include those details under the 'extraScrapeConfigs:|' section in the prometheus_values.yml file, as demonstrated here.



extraScrapeConfigs: |
  - job_name: 'api-svc'
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['api-svc.api-dev.svc.cluster.local:5557']
  - job_name: 'apps-svc'
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['apps-svc.api-dev.svc.cluster.local:5559']

Run Helm command to install Prometheus (for worker node's AZ storage class option):



helm upgrade -i prometheus -f prometheus_values.yml prometheus-community/prometheus \
  --namespace prometheus --version 15

Run Helm command to install Prometheus (for default storage class option):



helm upgrade -i prometheus -f prometheus_values.yml prometheus-community/prometheus \
  --namespace prometheus \
  --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2" \
  --version 15

Important Note: The default storage class has a reclaim policy set to "Delete" Consequently, any EBS volumes used by Prometheus will be automatically deleted when you remove Prometheus itself.

Once Helm installation is completed, let's verify the resources.



$ k get all -n prometheus
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/prometheus-alertmanager-c7644896-7kfjq           2/2     Running   0          103m
pod/prometheus-kube-state-metrics-8476bdcc64-wng4m   1/1     Running   0          103m
pod/prometheus-node-exporter-8hf57                   1/1     Running   0          103m
pod/prometheus-pushgateway-665779d98f-v8q5d          1/1     Running   0          103m
pod/prometheus-server-6fd8bc8576-wwmvw               2/2     Running   0          103m

NAME                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/prometheus-alertmanager         ClusterIP   172.20.84.40     <none>        80/TCP         103m
service/prometheus-kube-state-metrics   ClusterIP   172.20.192.129   <none>        8080/TCP       103m
service/prometheus-node-exporter        ClusterIP   None             <none>        9100/TCP       103m
service/prometheus-pushgateway          ClusterIP   172.20.181.13    <none>        9091/TCP       103m
service/prometheus-server               NodePort    172.20.167.19    <none>        80:30900/TCP   103m

NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-node-exporter   1         1         1       1            1           <none>          103m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-alertmanager         1/1     1            1           103m
deployment.apps/prometheus-kube-state-metrics   1/1     1            1           103m
deployment.apps/prometheus-pushgateway          1/1     1            1           103m
deployment.apps/prometheus-server               1/1     1            1           103m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-alertmanager-c7644896           1         1         1       103m
replicaset.apps/prometheus-kube-state-metrics-8476bdcc64   1         1         1       103m
replicaset.apps/prometheus-pushgateway-665779d98f          1         1         1       103m
replicaset.apps/prometheus-server-6fd8bc8576               1         1         1       103m

The chart creates two persistent volume claims: an 8Gi volume for prometheus-server pod and a 2Gi volume for prometheus-alertmanager.

iv. Check metrics from Prometheus

To inspect metrics from Prometheus in the browser, you must initiate port forwarding.



kubectl port-forward -n prometheus deploy/prometheus-server 8081:9090 &

Now, open a web browser and navigate to http://localhost:8081/targets

From the page you can see all configured metrics, alerts, rules & other configurations.

4. Install Grafana

In this step, we will create a dedicated Kubernetes namespace for Grafana, create the Grafana manifest file, setup security groups, setup an Ingress, and finally configure the dashboard.

i. Create a namespace for Grafana

Create a namespace called 'grafana'.
kubectl create namespace grafana

ii. Create a Grafana mainfest file

We also require a manifest file to configure Grafana. Below is an example of the file named grafana.yaml.



# grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.prometheus.svc.cluster.local
        access: proxy
        isDefault: true

iii. Installing Grafana

Now, proceed to install Grafana using Helm. Replace 'my-password' with your password.



helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClass='gp2' \
    --set persistence.enabled=true \
    --set adminPassword='my-passoword' \
    --values grafana.yaml \
    --set service.type=NodePort

iv. Create a security group

Create a security group (grafana-alb-sg) with allowing https from anywhere as an inbound rule for ingress ALB.

v. Allow inbound request to EC2 worker node security group

Before exposing Grafana to the external world, let's examine the definition of the Kubernetes service responsible for running Grafana.



$ k -n grafana get svc grafana -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: grafana
  creationTimestamp: "2023-12-29T05:59:40Z"
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 10.2.2
    helm.sh/chart: grafana-7.0.14
  name: grafana
  namespace: grafana
  resourceVersion: "179748053"
  uid: 7da370e2-63a4-4ca6-8ad4-14e624a51c4f
spec:
  clusterIP: 172.20.102.0
  clusterIPs:
  - 172.20.102.0
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: service
    nodePort: 31059
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/name: grafana
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

The target port is set to 3000, which corresponds to the port utilized by pods running Grafana.

To enable inbound requests for port 3000, it is necessary to associate the security group created in the previous step with the EC2 worker nodes.

vi. Setup Ingress

Define a new Kubernetes Ingress to facilitate the provisioning of an ALB.

Additionally, assume that you have already installed the AWS Load Balancer Controller to enable Kubernetes Ingress in creating an ALB.

Let's define the Ingress definition file for Grafana.



apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: grafana
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/load-balancer-name: grafana-alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: ${PUBLIC_SUBNET_IDs}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/security-groups: ${ALB_SECURITY_GROUP_ID}
    alb.ingress.kubernetes.io/healthcheck-port: "3000"
    alb.ingress.kubernetes.io/healthcheck-path: /api/health
    alb.ingress.kubernetes.io/certificate-arn: ${ACM_CERT_ARN}
spec:
  rules:
    - host: ${YOUR_ROUTE53_DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 80

Replace values for subnets,secruity-groups,certificate-arn, host with your values.

Post applying the new Ingress and once the new ALB is ready, navigate to ${YOUR_ROUTE53_DOMAIN} to confirm that Grafana is now accessible.

After logging into your Grafana account, proceed to import the necessary dashboards.

You can download dashboards from Grafana Dashboards

I utilized these two dashboards, which proved to be valuable for monitoring the overall EKS Fargate cluster.

ID: 17119 Kubernetes EKS Cluster (Prometheus)
ID: 12421 Fargate Pod Requests

That concludes our walkthrough! In this guide, we established a new node group essential for Prometheus and Grafana, and successfully installed and configured both tools.

I trust this post proves valuable to you! 😊

Troubleshooting:
I've authored another post detailing additional issues and their solutions encountered during the setup of Prometheus & Grafana.

You can find the post here.

Additional references:

Troubleshooting: EKS + Helm + Prometheus + Grafana

Nowsath — Fri, 29 Dec 2023 12:06:26 +0000

In this post, I've compiled a list of issues along with their corresponding solutions that I encountered while configuring Prometheus and Grafana with Helm in the existing EKS Fargate cluster setup.

Error 1: Could not determine region from any metadata service. The region can be manually supplied via the AWS_REGION environment variable.

panic: did not find aws instance ID in node providerID string

$ k logs ebs-csi-controller-7f5c959c75-j92jf -n kube-system -c ebs-plugin
I1228 04:31:45.536047       1 driver.go:78] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.25.0"
I1228 04:31:45.536144       1 metadata.go:85] "retrieving instance data from ec2 metadata"
I1228 04:31:58.152468       1 metadata.go:88] "ec2 metadata is not available"
I1228 04:31:58.152491       1 metadata.go:96] "retrieving instance data from kubernetes api"
I1228 04:31:58.153081       1 metadata.go:101] "kubernetes api is available"
E1228 04:31:58.175387       1 controller.go:86] "Could not determine region from any metadata service. The region can be manually supplied via the AWS_REGION environment variable." err="did not find aws instance ID in node providerID string"
panic: did not find aws instance ID in node providerID string

$ kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-ebs-csi-driver,app.kubernetes.io/instance=aws-ebs-csi-driver"
NAME                                  READY   STATUS             RESTARTS       AGE
ebs-csi-controller-7f5c959c75-j92jf   0/6     CrashLoopBackOff   36 (9s ago)    10m
ebs-csi-controller-7f5c959c75-xpv9x   0/6     CrashLoopBackOff   36 (23s ago)   10m
ebs-csi-node-969qs                    3/3     Running            0              10m

Solution:
If you don't specify the region of your cluster when installing aws-ebs-csi-driver will result in the ebs-csi-controller pods crashing, as the default region will be set to 'us-east-1'.

helm upgrade --install aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.region=eu-north-1 \
  --set controller.serviceAccount.create=false \
  --set controller.serviceAccount.name=ebs-csi-controller-sa \
  aws-ebs-csi-driver/aws-ebs-csi-driver

This is because of the ebs-plugin container "Could not determine region from any metadata service. The region can be manually supplied via the AWS_REGION environment variable."

Error 2: Values don't meet the specifications of the schema(s) in the following chart(s)

Error: values don't meet the specifications of the schema(s) in the following chart(s):
prometheus:
- server.remoteRead: Invalid type. Expected: array, given: object
alertmanager:
- extraEnv: Invalid type. Expected: array, given: object

Solution:
The errors are a result of a version mismatch between the Prometheus version and the file used in the Helm installation. If you are using a customized prometheus_values.yml file, ensure you specify the precise version of Prometheus. Alternatively, if you do not use a customized file, make sure to use the latest version of the Prometheus file.

helm upgrade -i prometheus prometheus-community/prometheus \
  --namespace prometheus \
  --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2" \
  --version 15

I have used Prometheus version 15 here.

Error 3: 0/17 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/17 nodes are available: 17 Preemption is not helpful for scheduling..

$ k get events -n prometheus
LAST SEEN   TYPE      REASON               OBJECT                                                MESSAGE
2m13s       Warning   FailedScheduling     pod/prometheus-alertmanager-c7644896-td8xv            0/17 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/17 nodes are available: 17 Preemption is not helpful for scheduling..
47m         Normal    SuccessfulCreate     replicaset/prometheus-alertmanager-c7644896           Created pod: prometheus-alertmanager-c7644896-td8xv
2m30s       Warning   ProvisioningFailed   persistentvolumeclaim/prometheus-alertmanager         storageclass.storage.k8s.io "prometheus" not found

The pods, prometheus-alertmanager and prometheus-server, will remain in a pending status.

$ k get po -n prometheus
NAME                                             READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-c7644896-q2nzm           0/2     Pending   0          74s
prometheus-kube-state-metrics-8476bdcc64-f984p   1/1     Running   0          75s
prometheus-node-exporter-r82k7                   1/1     Running   0          74s
prometheus-pushgateway-665779d98f-zh2pf          1/1     Running   0          75s
prometheus-server-6fd8bc8576-csqt8               0/2     Pending   0          75s

Solution:
This is due to missing storage class of 'prometheus' as clearly shown in the events logs. So go ahead and create the storage class shown below.

EBS_AZ=$(kubectl get nodes \
  -o=jsonpath="{.items[0].metadata.labels['topology\.kubernetes\.io\/zone']}")

echo "
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: prometheus
  namespace: prometheus
provisioner: ebs.csi.aws.com
parameters:
  type: gp2
reclaimPolicy: Retain
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - $EBS_AZ
" | kubectl apply -f -

Error 4: Failed to provision volume with StorageClass "prometheus": rpc error: code = Internal desc = Could not create volume "pvc-48b7c3d8-d46a-47be-90e7-3d59eb3f5844": could not create volume in EC2: NoCredentialProviders: no valid providers in chain...

$ kubectl get events --sort-by=.metadata.creationTimestamp -n prometheus
LAST SEEN   TYPE      REASON                 OBJECT                                                MESSAGE
30s         Normal    Provisioning           persistentvolumeclaim/prometheus-alertmanager         External provisioner is provisioning volume for claim "prometheus/prometheus-alertmanager"
30s         Normal    Provisioning           persistentvolumeclaim/prometheus-server               External provisioner is provisioning volume for claim "prometheus/prometheus-server"
5s          Warning   ProvisioningFailed     persistentvolumeclaim/prometheus-alertmanager         failed to provision volume with StorageClass "prometheus": rpc error: code = Internal desc = Could not create volume "pvc-b7373f3b-3da9-47ac-8bfb-ad396816ce88": could not create volume in EC2: NoCredentialProviders: no valid providers in chain...
17s         Warning   ProvisioningFailed     persistentvolumeclaim/prometheus-server               failed to provision volume with StorageClass "prometheus": rpc error: code = Internal desc = Could not create volume "pvc-48b7c3d8-d46a-47be-90e7-3d59eb3f5844": could not create volume in EC2: NoCredentialProviders: no valid providers in chain...

Solution:
This issue arises from insufficient permissions assigned to the service account in the cluster, preventing it from provisioning the required persistent volumes.

You need to set service account details (with required IAM policies for the role) while install aws-ebs-csi-driver with Helm as shown here.

helm upgrade --install aws-ebs-csi-driver \
  --namespace kube-system \
  --set controller.region=eu-north-1 \
  --set controller.serviceAccount.create=false \
  --set controller.serviceAccount.name=ebs-csi-controller-sa \
  aws-ebs-csi-driver/aws-ebs-csi-driver

Error 5: The service account is absent in the EKS cluster setup, yet it is visible through the eksctl get command.

Solution:
Check whether you have added '--role-only' option while creating service account using eksctl.
If yes, delete the service account and recreate it without '--role-only' option as shown below.

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster api-dev \
  --role-name AmazonEKS_EBS_CSI_DriverRole \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --override-existing-serviceaccounts --approve

Here, 'api-dev' is the cluster name. Replace it with your cluster name before running the command.

Thank you for taking the time to read 👏😊! I will continue to update this post as I encounter new issues. Feel free to mention any unlisted issues in the comment section. 🤝❤️

Check my post on setting up Prometheus and Grafana with existing EKS Fargate cluster - Monitoring