DEV Community

Cover image for 🏗️ Building Production-Ready Highly Available Architecture on AWS: From Single Instance to Enterprise Scale [Week-4-P2] ☁️🚀
Suvrajeet Banerjee
Suvrajeet Banerjee Subscriber

Posted on • Edited on

🏗️ Building Production-Ready Highly Available Architecture on AWS: From Single Instance to Enterprise Scale [Week-4-P2] ☁️🚀

#devops #aws #git #cloud

🎯 Introduction: The Journey to Zero-Downtime Applications

In my previous blog post about AWS Week 4 fundamentals, I covered the foundational AWS services. This comprehensive guide takes you deeper into the advanced territory - building enterprise-grade, highly available applications that can withstand failures and scale automatically.

After completing the intensive Week 4 assignments of Pravin Mishra's DevOps Micro-Internship Cohort, I've gained hands-on experience with AWS's most critical high availability components. This blog breaks down the complex concepts into digestible insights, complete with real-world implementation details.

intro

🔧 What You'll Master by the End 🎓

This blog will transform your understanding of:

  • 🔥 Load Balancers - The intelligent traffic directors
  • 🔥 Auto Scaling Groups - Your application's dynamic scaling engine
  • 🔥 Target Groups - The health monitoring guardians
  • 🔥 Launch Templates - Standardized deployment blueprints
  • 🔥 AMIs (Amazon Machine Images) - Your application's DNA
  • 🔥 Multi-AZ Architecture - Geographically fault tolerant systems

Let's dive into each component and understand how they work together to create bulletproof applications.

💡 Understanding the Core Components: Building Blocks of High Availability 💡

🎯 Application Load Balancer (ALB): Your Traffic Control Tower 🎯

Think of an Application Load Balancer as the smartest traffic cop you've ever seen. But instead of re-directing cars, it's re-directs web requests to your application servers.

ALB

What ALB Actually Does:

  • 🌐 Intelligent Routing: Routes incoming requests to healthy instances only
  • 🌐 Health Monitoring: Continuously checks if your servers are responding properly
  • 🌐 SSL Termination: Handles HTTPS encryption/decryption
  • 🌐 Sticky Sessions: Can route users to the same server if needed

Real-World Scenario:

Imagine a popular restaurant with multiple dining rooms. The ALB is like the host who:

  • Checks which dining rooms have available tables (healthy instances)
  • Routes customers only to available rooms
  • Monitors if any room becomes full or unavailable
  • Never sends customers to closed dining rooms 
// ALB Configuration Example
{
  "LoadBalancerName": "EpicReads-ALB",
  "Scheme": "internet-facing",
  "Type": "application",
  "Listeners": [
    {
      "Protocol": "HTTP",
      "Port": 80,
      "DefaultActions": [
        {
          "Type": "forward",
          "TargetGroupArn": "arn:aws:elasticloadbalancing:region:account:targetgroup/EpicReads-TG"
        }
      ]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

🎯 Target Groups: The Health Check Specialists 🎯

Target Groups are like medical monitors in a hospital - they constantly check the vital signs of your application instances.

Core Functions:

  • 📊 Health Assessment: Sends HTTP requests to check instance health
  • 📊 Registration Management: Automatically adds/removes instances
  • 📊 Traffic Distribution: Only sends traffic to healthy instances
  • 📊 Port Mapping: Routes traffic to specific ports on instances

Health Check Process: 

# Target Group Health Check Flow
1. Send HTTP request to: http://instance-ip:8080/
2. Wait for response (timeout: 5 seconds)
3. Expect: HTTP 200 status code
4. Repeat every 30 seconds
5. Mark unhealthy after 2 consecutive failures
6. Mark healthy after 2 consecutive successes
Enter fullscreen mode Exit fullscreen mode

🔄 Auto Scaling Groups (ASG): Your Dynamic Instance Manager 🔄

Auto Scaling Groups are like having a super-intelligent facility manager who automatically hires or fires workers based on workload.

asg

Key Capabilities:

  • ⚡ Dynamic Scaling: Adds instances when CPU > 70%, removes when < 30%
  • ⚡ Health Replacement: Automatically replaces failed instances
  • ⚡ Multi-AZ Distribution: Spreads instances across availability zones
  • ⚡ Capacity Management: Maintains desired number of healthy instances

Scaling Scenarios: 

# Scenario 1: Traffic Spike (Black Friday Sale)
Current: 2 instances at 85% CPU
Action: Launch 2 more instances
Result: 4 instances at ~42% CPU each

# Scenario 2: Instance Failure
Current: 3 healthy instances, 1 failed
Action: Terminate failed instance, launch replacement
Result: 3 healthy instances maintained

# Scenario 3: Low Traffic (3 AM)
Current: 4 instances at 15% CPU
Action: Terminate 2 instances
Result: 2 instances at 30% CPU each
Enter fullscreen mode Exit fullscreen mode

📋 Launch Templates: Your Instance DNA Blueprint 📋

Launch Templates are like architectural blueprints - they define exactly how each new instance should be built.

Template Components:

  • 🔧 AMI Selection: Which operating system with pre-installed software
  • 🔧 Instance Type: Computing power (t2.micro, t3.medium, etc.)
  • 🔧 Security Groups: Network access rules against services
  • 🔧 User Data Script: Commands to run when instance starts
  • 🔧 Storage Configuration: Disk size and type 
#!/bin/bash
# User Data Script Example
cd /home/ubuntu/theepicbook
export NODE_ENV=production
npm install --production
nohup npm start > /home/ubuntu/app.log 2>&1 &
Enter fullscreen mode Exit fullscreen mode

🎭 AMIs (Amazon Machine Images): Your Application's Time Capsule 🎭

AMIs are like taking a perfect snapshot of your configured server - operating system, applications, configurations, and all.

AMI Creation Process:

  • 📸 Snapshot Creation: AWS creates an exact copy of your instance's storage
  • 📸 Configuration Capture: Includes all installed software and settings
  • 📸 Template Generation: Can be used to launch identical instances
  • 📸 Version Control: Multiple AMIs for different application versions

🌐 Traffic Flow: Following a User Request Through the Architecture 🌐

Let me hold your hand walk you through what happens when a user visits your highly available application:

Step-by-Step Request Journey: 

1. User types: http://epicreads-alb-1234567890.us-east-1.elb.amazonaws.com
   ↓
2. DNS resolves to ALB IP address
   ↓
3. ALB receives HTTP request
   ↓
4. ALB checks Target Group for healthy instances
   ↓
5. ALB selects Instance-A (80% healthy, lowest connections)
   ↓
6. Request forwarded to Instance-A:8080
   ↓
7. Nginx on Instance-A proxies to Node.js application
   ↓
8. Node.js queries RDS MySQL database
   ↓
9. Database returns book catalog data
   ↓
10. Node.js generates HTML response
    ↓
11. Response travels back through ALB to user
    ↓
12. User sees EpicBook homepage with book listings
Enter fullscreen mode Exit fullscreen mode

traffic-flow

Failure Scenario - What Happens When Things Go Wrong: 

# Instance Failure Detection
Target Group Health Check: FAILED
↓
Mark Instance-A as UNHEALTHY (30 seconds)
↓
Route new traffic ONLY to Instance-B and Instance-C
↓
Auto Scaling Group detects unhealthy instance
↓
Terminate Instance-A, launch replacement Instance-D
↓
Instance-D passes health checks (2 minutes)
↓
Add Instance-D to Target Group as HEALTHY
↓
Resume normal traffic distribution
Enter fullscreen mode Exit fullscreen mode

🔐 AWS Security Deep Dive: The Foundation Layer 🔐

Based on our previous discussion about AWS security architecture, let me explain where security groups fit into the infrastructure stack and why they're so critical for high availability.

The AWS Infrastructure Stack: 

1. Physical Data Centers (AWS-owned hardware)
   ↓
2. Hypervisor Layer (AWS Nitro System)
   ↓ 
3. 🔥 SECURITY GROUPS ENFORCED HERE 🔥
   ↓
4. Your EC2 Instance (Virtual Machine)
   ↓
5. Operating System (Ubuntu/Amazon Linux)
   ↓
6. Applications (Node.js, Nginx, MySQL)
Enter fullscreen mode Exit fullscreen mode

aws-infra-stack

Key Security Insights:

  • 🔒 Hardware-Level Enforcement: Security groups are implemented in specialized AWS Nitro Cards
  • 🔒 Pre-Instance Filtering: Traffic is blocked before reaching your virtual machine
  • 🔒 Immutable from Inside: You can't bypass security groups from within your instance
  • 🔒 Automatic Application: Rules apply instantly across all instances using that security group

Security Group Configuration for HA Architecture: 

# ALB Security Group (epicreads-alb-sg)
Inbound:
- HTTP (80) from 0.0.0.0/0 (Internet traffic)
- HTTPS (443) from 0.0.0.0/0 (Secure traffic)
Outbound:
- HTTP (8080) to Application Security Group

# Application Security Group (epicreads-app-sg)  
Inbound:
- SSH (22) from Admin IP (Management access)
- HTTP (8080) from ALB Security Group (App traffic)
Outbound:
- MySQL (3306) to Database Security Group
- HTTP/HTTPS (80/443) to 0.0.0.0/0 (Package updates)

# Database Security Group (epicreads-db-sg)
Inbound:  
- MySQL (3306) from Application Security Group
Outbound:
- None (Database doesn't initiate outbound connections)
Enter fullscreen mode Exit fullscreen mode

📊 Component Relationships: How Everything Works Together 📊

Here's how all the components integrate to create a resilient system:

The Integration Flow: 

graph TB
    A[Internet Users] --> B[Application Load Balancer]
    B --> C[Target Group]
    C --> D[Auto Scaling Group]
    D --> E[Launch Template]
    E --> F[AMI]
    D --> G[EC2 Instance 1]
    D --> H[EC2 Instance 2]  
    D --> I[EC2 Instance N]
    G --> J[RDS Multi-AZ]
    H --> J
    I --> J
Enter fullscreen mode Exit fullscreen mode

graph

Dependency Chain:

  1. AMI contains your application setup
  2. Launch Template references AMI + defines configuration
  3. Auto Scaling Group uses Launch Template to create instances
  4. Target Group monitors instance health
  5. ALB routes traffic based on Target Group health
  6. Multi-AZ RDS provides database high availability

integration-flow

🚀 Real-World Implementation: Building EpicBook's HA Architecture 🚀

Let me share the actual implementation details from my Week 4 assignments:

Phase 1: Foundation Setup 

# 1. VPC Architecture
VPC: 10.0.0.0/16
├── PublicSubnet1: 10.0.0.0/24 (ap-south-1a)
├── PublicSubnet2: 10.0.1.0/24 (ap-south-1b)  
├── PrivateSubnet1: 10.0.2.0/24 (ap-south-1a)
└── PrivateSubnet2: 10.0.3.0/24 (ap-south-1b)

# 2. ALB Configuration
Name: EpicReads-ALB
Scheme: Internet-facing
Subnets: PublicSubnet1, PublicSubnet2
Security Group: epicreads-alb-sg
Enter fullscreen mode Exit fullscreen mode

Phase 2: Auto Scaling Implementation 

# 1. Custom AMI Creation
Base Image: Ubuntu 22.04 LTS
Pre-installed: Node.js 20.x, Nginx, Git
Application: EpicBook source code
Configuration: Production environment

# 2. Launch Template
Name: EpicReads-Launch-Template
AMI: EpicReads-App-AMI-v1
Instance Type: t2.micro
Security Group: epicreads-app-sg
User Data: Application startup script

# 3. Auto Scaling Group
Name: EpicReads-ASG
Launch Template: EpicReads-Launch-Template
Min: 2 instances, Max: 6 instances, Desired: 2
AZs: ap-south-1a, ap-south-1b
Target Group: EpicReads-TG
Enter fullscreen mode Exit fullscreen mode

Phase 3: Database High Availability 

# Multi-AZ RDS Configuration
Engine: MySQL 8.0
Instance: db.t3.micro
Multi-AZ: Enabled (Primary in ap-south-1a, Standby in ap-south-1b)
Subnet Group: epicreads-db-subnet-group
Security Group: epicreads-db-sg
Automated Backups: 7 days retention
Enter fullscreen mode Exit fullscreen mode

⚡ Performance Optimization & Best Practices ⚡

Load Balancer Optimization:

🎯 Health Check Tuning:

# Optimal Health Check Settings
Path: /health (custom endpoint)
Interval: 30 seconds
Timeout: 5 seconds
Healthy Threshold: 2
Unhealthy Threshold: 5
Enter fullscreen mode Exit fullscreen mode

🎯 Connection Draining:

# Graceful Instance Removal
Deregistration Delay: 300 seconds
# Allows existing connections to complete
# Prevents connection drops during scaling
Enter fullscreen mode Exit fullscreen mode

Auto Scaling Best Practices:

⚙️ Scaling Policies:

# Scale Out Policy
Metric: Average CPU Utilization > 70%
Cooldown: 300 seconds
Scaling Adjustment: +1 instance

# Scale In Policy  
Metric: Average CPU Utilization < 30%
Cooldown: 300 seconds
Scaling Adjustment: -1 instance
Enter fullscreen mode Exit fullscreen mode

⚙️ Instance Warmup:

# Application Startup Time
Instance Launch: ~2 minutes
Application Start: ~30 seconds
Health Check Pass: ~1 minute
Total Ready Time: ~3.5 minutes
Enter fullscreen mode Exit fullscreen mode

🔥 Advanced Troubleshooting: Common Issues & Solutions 🔥

⏩ Issue 1: 502 Bad Gateway Error

Symptoms: ALB returns 502 error to users
Root Cause: Application not responding on configured port
Solution:

# Check application status
sudo systemctl status nginx
sudo netstat -tlnp | grep :8080

# Fix application startup
cd /home/ubuntu/theepicbook
npm start

# Update health check path
Target Group → Health Checks → Edit
Path: /health (create custom health endpoint — returning "OK" status)
Enter fullscreen mode Exit fullscreen mode

⏩ Issue 2: Auto Scaling Not Triggering

Symptoms: High CPU but no new instances launching

Root Cause: CloudWatch metrics not being collected
Solution:

# Install CloudWatch agent
sudo yum install amazon-cloudwatch-agent
sudo systemctl start amazon-cloudwatch-agent

# Verify scaling policies
Auto Scaling Group → Automatic Scaling → View scaling policies
Check: Target value, cooldown periods, metric collection
Enter fullscreen mode Exit fullscreen mode

⏩ Issue 3: Database Connection Failures

Symptoms: Application can't connect to RDS
Root Cause: Security group misconfiguration
Solution:

# Check security group rules
RDS Security Group Inbound Rules:
✅ MySQL (3306) from Application Security Group
❌ NOT from 0.0.0.0/0 (security risk)

# Verify connection string
- Host: epicreads-database.cpmao2e6cx2i.ap-south-1.rds.amazonaws.com
- Port: 3306
- Database: bookstore
Enter fullscreen mode Exit fullscreen mode

📈 Cost Optimization Strategies: Maximum Efficiency 📈

Right-Sizing Your Architecture:

💰 Instance Selection:

# Development Environment
ALB: 1 ALB unit (~$16/month)
EC2: 2 × t2.micro (~$17/month)
RDS: 1 × db.t3.micro (~$20/month)
Total: ~$53/month

# Production Environment  
ALB: 1 ALB unit (~$16/month)
EC2: 2-6 × t3.medium (~$140-420/month)
RDS: 1 × db.t3.small Multi-AZ (~$80/month)
Total: ~$236-516/month
Enter fullscreen mode Exit fullscreen mode

architecture

💰 Cost Optimization Techniques:

  • Use Reserved Instances for baseline capacity (save up to 75%)
  • Implement Spot Instances for fault-tolerant workloads
  • Enable Auto Scaling to avoid over-provisioning
  • Use CloudWatch to monitor and optimize resource usage

🎯 Testing High Availability: Proving Your Architecture Works 🎯

Disaster Recovery Testing:

🧪 Test Scenario 1: Instance Failure

# Simulate instance failure
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0

# Expected Results:
- ALB stops routing traffic to failed instance (30 seconds)
- Auto Scaling Group launches replacement (2-3 minutes)
- Application remains accessible throughout
- Zero downtime for users
Enter fullscreen mode Exit fullscreen mode

🧪 Test Scenario 2: AZ Failure

# Simulate availability zone failure
# Terminate all instances in ap-south-1a

# Expected Results:
- Traffic routes to instances in ap-south-1b
- Auto Scaling Group launches instances in healthy AZ
- RDS fails over to standby (if primary AZ affected)
- Application maintains availability
Enter fullscreen mode Exit fullscreen mode

fault-tolerance

🧪 Test Scenario 3: Load Testing

# Generate traffic spike
ab -n 10000 -c 100 http://epicreads-alb-xxx.elb.amazonaws.com/

# Expected Results:
- CPU utilization increases beyond 70%
- Auto Scaling Group launches additional instances
- ALB distributes load across all healthy instances
- Response times remain acceptable
Enter fullscreen mode Exit fullscreen mode

🚀 Next Level: Advanced Features & Future Enhancements 🚀

Enhanced Security:

  • 🔒 AWS WAF Integration: Block malicious traffic
  • 🔒 SSL/TLS Certificates: HTTPS encryption
  • 🔒 IAM Roles: Secure service-to-service communication

Performance Improvements:

  • ⚡ CloudFront CDN: Global content delivery
  • ⚡ ElastiCache: In-memory caching layer
  • ⚡ RDS Read Replicas: Read traffic distribution

Operational Excellence:

  • 📊 CloudWatch Dashboards: Custom monitoring
  • 📊 AWS Systems Manager: Centralized instance management
  • 📊 AWS Config: Configuration compliance tracking

🎓 Key Takeaways: Your High Availability Mastery Checklist 🎓

  • ✅ Load Balancers distribute traffic intelligently and ensure high availability
  • ✅ Auto Scaling Groups provide dynamic scaling and automatic failure recovery
  • ✅ Target Groups monitor instance health and manage traffic routing
  • ✅ Launch Templates standardize instance deployments for consistency
  • ✅ AMIs capture your application state for rapid deployment
  • ✅ Multi-AZ Architecture provides geographic fault tolerance
  • ✅ Security Groups enforce network security at the hypervisor level
  • ✅ End-to-end testing validates your architecture's resilience

🎯 Conclusion: From Single Point of Failure to Enterprise Grade 🎯

Through this comprehensive journey, you've learned how to transform a simple single-instance application into a production-ready, highly available system that can handle real-world challenges.

spof

The EpicBook application we built demonstrates enterprise-grade architecture patterns:

  • Zero single points of failure
  • Automatic scaling and recovery
  • Geographic distribution
  • Comprehensive monitoring
  • Security best practices

This marks the completion of an intensive learning journey through AWS high availability architecture - a crucial skillset for any DevOps engineer building production systems.

🚀 What's Next? Continue Your DevOps Journey

This comprehensive deep-dive concludes my Week 4 documentation for the DevOps Micro-Internship Cohort. The concepts covered here form the foundation for building resilient & scalable applications in the cloud.

🔗 Connect & Continue Learning

P.S. This post marks the completion of Week 4 in the DevOps Micro-Internship Cohort run by Pravin Mishra 🙏. This intensive hands-on experience has been transformational for understanding enterprise cloud architecture.

You can start your DevOps journey for free from Pravin's comprehensive YouTube Playlist.

🏷️ Series Navigation

This blog is part of my DevOps Series documenting the complete journey from fundamentals to advanced implementations.

Previous: AWS Week 4 - Foundations

Next: Week 5 — Infrastructure as Code (IaC) with AWS CloudFormation (Coming Soon...) 😎😁😵

Building resilient systems requires understanding each component in depth. This continuos posting, sharing and documenting my knowledge base in form of blog posts, Linkedin posts serves as a reference for future implementations and continuous learning in the ever-evolving world of cloud infrastructure.

Happy building! 🏗️⚡

🏷️ Tags:

#AWS #DevOps #CloudComputing #HighAvailability #CloudArchitecture #LoadBalancing #AutoScaling #ApplicationLoadBalancer #TargetGroups #LaunchTemplates #AMI #MultiAZ #EC2 #RDS #VPC #SecurityGroups #FaultTolerance #ScalableArchitecture #CloudInfrastructure #ProductionDeployment #NetworkSecurity #DatabaseDesign #InfrastructureAsCode #TechEducation #EnterpriseArchitecture #DisasterRecovery #CloudOptimization #DevOpsEngineering #SystemAdministration #NodeJS #MySQL #Nginx #Ubuntu #TechSkills #LearningByDoing #DevOpsJourney #CloudFormation #Monitoring #CostOptimization #LearnInPublic #DevOpsLife #TechCommunity #Mentorship #CloudDeployment #WebDev #SysAdmin #TeamWork #Programming #TechBlog #AWSCertification #CloudSecurity #AutomatedScaling #LoadBalancer #DatabaseHA #NetworkArchitecture #ProductionReadySystem #ZeroDowntime #EnterpriseGrade #CloudBestPractices #AWSServices #TechnicalWriting #CloudMastery #ArchitecturalPatterns #SystemDesign #CloudStrategy #ProfessionalDevelopment #TechInsights #CloudExpertise #AdvancedAWS #CloudNative #ModernInfrastructure

Top comments (0)