DEV Community

Cover image for Production-Grade 3-Tier Image Processing Platform
Ntseze-Nelvis
Ntseze-Nelvis

Posted on

Production-Grade 3-Tier Image Processing Platform

#cloud #python #security #serverless

==========================================================================

HSBC-gamma:Production-Grade 3-Tier Image Processing Platform

A secure, scalable, and highly available image processing platform built on AWS with Terraform

πŸ“‹ Overview

This project implements a production-grade 3-tier image processing platform on AWS using Infrastructure as Code (IaC) with Terraform. Users can upload images through a web interface, which are then automatically processed (resized into multiple versions) and stored securely in encrypted S3 buckets.

Key Features

  • 53 Terraform resources deployed in AWS
  • Multi-AZ high availability across eu-north-1a and eu-north-1b
  • Auto-scaling (1-3 instances per tier based on CPU utilization)
  • KMS encryption for all S3 data at rest
  • Least privilege IAM with instance profiles (no access keys on EC2)
  • Path-based routing with Application Load Balancer
  • Complete observability with CloudWatch logs and alarms
  • Self-healing architecture with health checks

πŸ“‚ Project Structure 

C:.
β”‚   .gitignore
β”‚   .gitlab-ci.yml
β”‚   .terraform.lock.hcl
β”‚   complete-configuration.txt
β”‚   elbv2 describe-target-groups _
β”‚   elbv2 describe-target-groups 
β”‚   main.tf
β”‚   outputs.tf
β”‚   print_project.sh
β”‚   project_tracker.sh
β”‚   providers.tf
β”‚   README.md
β”‚   terraform.tfstate
β”‚   terraform.tfstate.backup
β”‚   terraform.tfvars
β”‚   variables.tf
β”‚
β”œβ”€β”€β”€.terraform
β”‚   β”œβ”€β”€β”€modules
β”‚   β”‚       modules.json
β”‚   β”‚
β”‚   └───providers
β”‚       └───registry.terraform.io
β”‚           └───hashicorp
β”‚               └───aws
β”‚                   └───6.31.0
β”‚                       └───windows_amd64
β”‚                               LICENSE.txt
β”‚                               terraform-provider-aws_v6.31.0_x5.exe
β”‚
└───modules
    β”œβ”€β”€β”€alb
    β”‚       main.tf
    β”‚       outputs.tf
    β”‚       variables.tf
    β”‚
    β”œβ”€β”€β”€app
    β”‚       asg.tf
    β”‚       launch_template.tf
    β”‚       outputs.tf
    β”‚       user_data.sh
    β”‚       variables.tf
    β”‚
    β”œβ”€β”€β”€monitoring
    β”‚       alarms.tf
    β”‚       cloudwatch.tf
    β”‚       logs.tf
    β”‚       user_data.sh
    β”‚       variables.tf
    β”‚
    β”œβ”€β”€β”€s3
    β”‚       buckets.tf
    β”‚       kms.tf
    β”‚       lifecycle.tf
    β”‚       outputs.tf
    β”‚       test-upload.html
    β”‚       variables.tf
    β”‚
    β”œβ”€β”€β”€security
    β”‚       iam.tf
    β”‚       outputs.tf
    β”‚       sg.tf
    β”‚       variables.tf
    β”‚
    β”œβ”€β”€β”€vpc
    β”‚       main.tf
    β”‚       outputs.tf
    β”‚       variables.tf
    β”‚
    └───web
            asg.tf
            launch_template.tf
            output.tf
            user_data.sh
            variables.tf
Enter fullscreen mode Exit fullscreen mode

Module Resource Count

Module Purpose Resources
vpc Network infrastructure 11
security Security controls 12
s3 Storage layer 9
alb Load balancing 4
web Frontend tier 3
app Backend tier 3
monitoring Observability 7
Total 53

Architecture Diagram

HSBC-Gamma-Achitectural-Diagram
HSBC-Gamma-Achitectural-Diagram

Quick Start

Prerequisites 

# AWS CLI installed and configured
aws configure
# Enter your credentials and region: eu-north-1

# Terraform v1.0+ installed
terraform --version

# Clone the repository
git clone https://github.com/Ntseze-Nelvis/hsbc-gamma-3tier-image-platform.git
cd hsbc-gamma-3tier-image-platform-main
Enter fullscreen mode Exit fullscreen mode

Deploy Infrastructure 

# Initialize Terraform
terraform init

# Format and validate configuration
terraform fmt -recursive
terraform validate

# Review what will be created
terraform plan
**Terraform plan**
![Terraform-plan](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w6g8px681qz6oe41w8gs.jpg)

# Deploy infrastructure
terraform apply -auto-approve

## View outputs
terraform output
Enter fullscreen mode Exit fullscreen mode

Expected Outputs 

alb_dns_name = "hsbc-gamma-dev-alb-50807543.eu-north-1.elb.amazonaws.com"
app_asg_name = "app-asg"
kms_key_arn = "arn:aws:kms:eu-north-1:211125430491:key/bb2beb00-4920-44e4-8ea6-6fb2a554b0e8"
processed_bucket_arn = "arn:aws:s3:::hsbc-gamma-dev-processed-images"
raw_bucket_arn = "arn:aws:s3:::hsbc-gamma-dev-raw-images"
web_asg_name = "web-asg"
Enter fullscreen mode Exit fullscreen mode

Terraform apply
Terraform-apply

Test Your Application

Open your browser and navigate to: 

http://hsbc-gamma-dev-alb-50807543.eu-north-1.elb.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Upload and Process an Image
Click "Choose Image" or drag & drop an image

Wait 5-10 seconds for processing

HSBC-Gamma-Web-App

HSBC-Gamma-Web-App

Verify Processing 

## Check raw bucket
aws s3 ls s3://hsbc-gamma-dev-raw-images/

## Check processed bucket
aws s3 ls s3://hsbc-gamma-dev-processed-images/
Enter fullscreen mode Exit fullscreen mode
  • Expected output shows:
  • thumbnail-uuid.jpg
  • small-uuid.jpg
  • medium-uuid.jpg
  • large-uuid.jpg

Monitoring & Observability

View CloudWatch Logs 

# App tier logs (Flask API)
aws logs tail /hsbc-gamma/app --since 1h --follow

# Web tier logs (Apache)
aws logs tail /hsbc-gamma/web --since 1h --follow
Enter fullscreen mode Exit fullscreen mode

Check Auto Scaling Status 

# Web tier instances
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names web-asg

## App tier instances
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names app-asg
Enter fullscreen mode Exit fullscreen mode

Monitor ALB Health 

# Check target group health
aws elbv2 describe-target-health \
  --target-group-arn $(aws elbv2 describe-target-groups \
    --names hsbc-gamma-dev-alb-web-tg \
    --query 'TargetGroups[0].TargetGroupArn' \
    --output text)
    ```
{% endraw %}

## Security Validation
## Verify S3 Encryption
{% raw %}

```bash
aws s3api get-bucket-encryption --bucket hsbc-gamma-dev-raw-images
# Expected: KMS encryption enabled
Enter fullscreen mode Exit fullscreen mode

Verify Public Access Block 

aws s3api get-public-access-block --bucket hsbc-gamma-dev-raw-images
# Expected: All blocks enabled
Enter fullscreen mode Exit fullscreen mode

Verify Security Groups 

aws ec2 describe-security-groups --group-names hsbc-gamma-dev-web-sg \
  --query 'SecurityGroups[0].IpPermissions'
# Expected: Only port 80 from ALB security group
Enter fullscreen mode Exit fullscreen mode

Sample Calculation (2 instances each tier)

EC2: 4 Γ— $8.76 = $35.04
ALB: $0.0225/h Γ— 730h = $16.43
NAT: $0.045/h Γ— 730h = $32.85
S3: Estimated = $8.00
KMS: $1.00 + operations = $1.50
CW: Logs + metrics = $4.00
Data: Estimated = $15.00
─────────────────────────────────
Total: $112.82/month

Troubleshooting

Issue 1: Can't Access Web UI 

# Check ALB DNS
terraform output alb_dns_name

## Verify ALB is active
aws elbv2 describe-load-balancers --names hsbc-gamma-dev-alb \
  --query 'LoadBalancers[0].State.Code'

## Test ALB endpoint
curl -I http://$(terraform output -raw alb_dns_name)
Enter fullscreen mode Exit fullscreen mode

Verify ALB is in active state
Ensure security group allows port 80 from 0.0.0.0/0
Check that web tier instances are healthy

Issue 2: Image Upload Fails

Symptoms: Upload returns server error, image not processed

Diagnostic Commands: 

# Check app tier logs for errors
aws logs tail /hsbc-gamma/app --since 30m --filter-pattern "ERROR"

# Verify target group health
aws elbv2 describe-target-health \
  --target-group-arn $(aws elbv2 describe-target-groups \
    --names hsbc-gamma-dev-alb-app-tg \
    --query 'TargetGroups[0].TargetGroupArn' \
    --output text)
    #Diagnostic Commands:
Enter fullscreen mode Exit fullscreen mode

Common Solutions:

  • Restart Flask app on app tier instances
  • Verify IAM role has S3 permissions
  • Check S3 bucket policies allow app role access 
# Issue 3: Instances Not Launching
#Symptoms: Auto Scaling group shows 0 instances or desired capacity not met
#Diagnostic Commands:

## Check ASG status
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names web-asg

## View scaling activities
aws autoscaling describe-scaling-activities --auto-scaling-group-name web-asg
Enter fullscreen mode Exit fullscreen mode

Clean UpClean Up 

# Destroy all infrastructure
terraform destroy -auto-approve

# Verify resources are deleted
aws s3 ls | grep hsbc-gamma-dev
aws ec2 describe-instances --filters "Name=tag:Name,Values=*hsbc*"
aws elbv2 describe-load-balancers --names hsbc-gamma-dev-alb
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names web-asg app-asg
Enter fullscreen mode Exit fullscreen mode

Resources & Documentation

Resource Link
Terraform AWS Provider registry.terraform.io/providers/hashicorp/aws
AWS 3-Tier Architecture aws.amazon.com/architecture/3-tier
IAM Instance Profiles docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html
Flask Documentation flask.palletsprojects.com
Pillow Documentation pillow.readthedocs.io
Auto Scaling Groups docs.aws.amazon.com/autoscaling/ec2/userguide/
Application Load Balancer docs.aws.amazon.com/elasticloadbalancing/latest/application/

Terraform Operations 

terraform init                                      # Initialize
terraform validate                                  # Validate config
terraform plan                                      # Preview changes
terraform apply -auto-approve                       # Deploy
terraform destroy -auto-approve                     # Destroy all
terraform output                                    # View outputs
terraform output -raw alb_dns_name                  # Get specific output
terraform state list                                # List resources
terraform state show aws_instance.example          # Show resource details
terraform graph | dot -Tpng > graph.png            # Generate graph

# State Management
terraform state mv aws_instance.old aws_instance.new # Move resource
terraform state rm aws_instance.to_remove           # Remove from state
terraform import aws_instance.example i-12345678    # Import resource
Enter fullscreen mode Exit fullscreen mode

AWS CLI Commands 

# Account & Identity
aws sts get-caller-identity                        # Verify account

# EC2 & Compute
aws ec2 describe-instances --filters "Name=tag:Name,Values=*hsbc*"

# Auto Scaling
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names web-asg
aws autoscaling describe-scaling-activities --auto-scaling-group-name web-asg

# Load Balancer
aws elbv2 describe-load-balancers --names hsbc-gamma-dev-alb
aws elbv2 describe-target-groups --names hsbc-gamma-dev-alb-web-tg

# S3 Storage
aws s3 ls s3://hsbc-gamma-dev-raw-images/ --recursive --summarize
aws s3api get-bucket-encryption --bucket hsbc-gamma-dev-raw-images

# CloudWatch
aws logs tail /hsbc-gamma/app --since 1h --follow
aws logs filter-log-events --log-group-name /hsbc-gamma/app --filter-pattern "ERROR"
Enter fullscreen mode Exit fullscreen mode

Monitoring & Debugging 

# Real-time monitoring
watch -n 5 'aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names web-asg --query "AutoScalingGroups[0].Instances[*].LifecycleState"'

# Check ASG instances health
watch -n 10 'aws elbv2 describe-target-health --target-group-arn $(aws elbv2 describe-target-groups --names hsbc-gamma-dev-alb-web-tg --query "TargetGroups[0].TargetGroupArn" --output text)'

# Network connectivity test
curl -w "Connect: %{time_connect}s TTFB: %{time_starttransfer}s Total: %{time_total}s\n" -o /dev/null -s http://$(terraform output -raw alb_dns_name)
Enter fullscreen mode Exit fullscreen mode

License

This project is licensed under the MIT License - see the LICENSE file for details.

Author

Owner : NTSEZE VOUFFO NELVIS

Role Responsibilities

Role Responsibilities
Solutions Architect Architecture design, security compliance, system diagrams, trade-off analysis
Cloud/DevOps Engineer Infrastructure automation, CI/CD, deployment, monitoring setup

Top comments (0)