DEV Community: Manish Kumar

From Zero to Production AWS in One `terraform apply`: A Look at terraform-aws-infrastructure

Manish Kumar — Tue, 08 Sep 2026 16:09:19 +0000

If you've ever spent an afternoon clicking through the AWS console to wire up a VPC, subnets, an ALB, TLS certificates, and a CloudFront distribution — only to realize you can't remember exactly what you clicked — you already understand the problem manishpcp/terraform-aws-infrastructure sets out to solve.

This is a fully modular, production-oriented Terraform project that codifies an entire multi-tier AWS environment: private compute, a public load balancer, a CDN-backed static site, a containerized microservice, DNS, observability, and CI/CD — all wired together and deployable with a single command.

The Problem It Solves

Manually building secure, observable AWS infrastructure has a few recurring pain points, and this project tackles each one directly:

Infrastructure drift. Console changes made by different people over time quietly diverge across environments. Here, every resource is declared in code, reviewed in Git, and applied deterministically.
Overexposed compute. It's common to see EC2 instances sitting directly on the public internet. In this architecture, compute lives in private subnets with no public IPs — only the ALB and NAT Gateways face the internet.
Certificate babysitting. ACM certificates for both the ALB and CloudFront are provisioned and DNS-validated automatically, removing manual renewal overhead.
Serving static and dynamic workloads from one domain. Many teams need a static frontend and a dynamic API under the same domain. This stack handles both — CloudFront + S3 for static content, ALB + EC2/Docker for the API — under a single Route 53 hosted zone.
Missing observability. Fresh EC2 instances typically ship with no logs or metrics. Here, the CloudWatch Agent is installed and configured at boot on every instance automatically.
Long-lived CI/CD credentials. Instead of storing AWS keys in GitHub, both pipelines authenticate via OIDC, assuming an IAM role with no static credentials anywhere.
Unbounded container image growth. An ECR lifecycle policy automatically retains only the last 10 images, keeping storage costs in check.

The Architecture

The stack spans two Availability Zones for resilience and separates concerns cleanly between public and private tiers:

A VPC (10.0.0.0/16) with 2 public and 2 private subnets, each AZ getting its own NAT Gateway.
Two EC2 instances (t3.small, Amazon Linux 2023) in private subnets, each running Nginx and a Docker container side by side, with encrypted gp3 volumes and IMDSv2 enforced.
An internet-facing Application Load Balancer with HTTP→HTTPS redirect, TLS 1.3, and host-header-based routing to separate Nginx and Docker target groups.
S3 + CloudFront for the static site, with Origin Access Control so the bucket is never publicly reachable directly, plus geo-restriction and gzip compression.
ECR hosting a Flask/Gunicorn microservice with /, /health, and /info endpoints, built and pushed via GitHub Actions.
Route 53 automatically wiring up seven subdomains across the direct EC2 endpoints, the load-balanced endpoints, and the static site.
CloudWatch log groups and agent-shipped metrics (CPU, memory, disk) from every instance.
A least-privilege IAM instance profile scoped to CloudWatch, SSM Parameter Store, and ECR read access.

The Technology Stack

Terraform (>= 1.5.0) with a modular layout — separate modules for VPC, security groups, IAM, EC2, ALB, CloudWatch, Route 53, and S3/CloudFront — plus S3 + DynamoDB for remote state.
AWS services: VPC, EC2, Application Load Balancer, ACM, S3, CloudFront, Route 53, ECR, CloudWatch, IAM, SSM.
Nginx as a reverse proxy on each instance, routing subdomain traffic to either a static response or the local Docker container.
Docker running the containerized workload on each instance.
Python (Flask + Gunicorn) powering the microservice deployed through ECR.
GitHub Actions for CI/CD, using OIDC federation to assume an IAM role rather than storing AWS credentials as secrets — one workflow handles terraform plan/apply, the other builds and pushes the Docker image.

Why It's Worth a Look

What stands out about this repo isn't just that it automates AWS provisioning — plenty of projects do that. It's that it demonstrates a genuinely production-shaped pattern: private-by-default compute, automatic TLS, OIDC-based CI/CD instead of static keys, built-in observability, and a real cost breakdown (roughly $125–135/month at low traffic, with NAT Gateways called out as the biggest line item and a suggestion to consolidate them for non-prod environments).

The README also includes an honest security review — flagging an outdated Gunicorn version vulnerable to request smuggling, a Flask cache header issue, S3 public access block settings that need tightening, and a couple of ALB hardening steps. That kind of transparency about what still needs fixing before real production use is refreshingly rare in reference infrastructure projects.

If you're learning Terraform module design, looking for a realistic multi-tier AWS reference architecture, or just want a batteries-included starting point for a hybrid static-site-plus-microservice setup, this repository is a solid one to study — and to fork.

Repo: github.com/manishpcp/terraform-aws-infrastructure

Building Enterprise AWS Networks: A Complete Guide to VPC Sharing, Transit Gateway, Private NAT Gateway, and AWS Organizations

Manish Kumar — Thu, 25 Jun 2026 15:24:49 +0000

Introduction

In modern enterprise cloud architectures, organizations increasingly adopt multi-account strategies to improve security, isolate workloads, manage billing, and enable team autonomy. However, this approach introduces complex networking challenges that require careful planning. This blog explores how VPC Sharing, AWS Transit Gateway, Private NAT Gateway, and AWS Organizations work together to create a standardized, scalable, and secure enterprise multi-account networking pattern.

Understanding this integration is critical for cloud architects, DevOps engineers, and infrastructure teams managing large-scale AWS deployments. Without these components, organizations face overlapping CIDR blocks, uncontrolled internet egress, complex VPC peering configurations, and lack of centralized governance—leading to security vulnerabilities, operational inefficiencies, and cost overruns.

Why Multi-Account Architecture Matters

Key Benefits

Benefit	Description
Security Isolation	Separate development, testing, staging, and production environments into distinct accounts
Billing Transparency	Track costs per department, project, or team without manual aggregation
Access Control	Apply granular IAM policies and SCPs per account or organizational unit
Workload Isolation	Prevent accidental cross-contamination between critical and non-critical systems
Team Autonomy	Enable teams to provision resources independently while maintaining governance guardrails

However, multi-account architectures introduce networking complexity. Traditional VPC peering becomes unmanageable at scale (requiring N×(N-1)/2 connections), CIDR overlaps prevent peering, and each account needs its own NAT gateway—increasing costs and operational overhead.

Component 1: AWS Organizations

What It Is

AWS Organizations provides centralized governance across multiple AWS accounts. It enables:

Account Creation & Management: Streamlined provisioning of new accounts
Organizational Units (OUs): Group accounts by function (e.g., Infrastructure, Production, Development)
Service Control Policies (SCPs): Define permission guardrails that apply to all accounts in an OU
All Features Enablement: Required for advanced capabilities like SCPs, AWS RAM, and centralized governance

Enterprise OU Structure Example

o-1234567890abcdef (Root Organization)
├── Infrastructure_Prod (OU)
│   ├── network-prod (Account: Central networking with Transit Gateway)
│   └── security-prod (Account: Network Firewall, WAF)
├── Production (OU)
│   ├── app-prod-1 (Account: Production application workloads)
│   └── app-prod-2 (Account: Production application workloads)
├── Development (OU)
│   ├── app-dev-1 (Account: Development workloads)
│   └── app-dev-2 (Account: Development workloads)
└── Sandbox (OU)
    └── sandbox-1 (Account: Experimental workloads)

Why You Need AWS Organizations

Without AWS Organizations:

No centralized account governance
Manual permission management across accounts
Inconsistent security policies
Impossible to use AWS RAM for resource sharing across accounts
No SCP-based guardrails to prevent costly or insecure actions

Key SCP Examples

Example 1: Prevent External Resource Sharing

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": [
        "ram:CreateResourceShare",
        "ram:AssociateResourceShare"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalOrgID": "o-1234567890"
        }
      }
    }
  ]
}

This SCP prevents accounts from sharing resources outside your organization, ensuring all resource sharing stays within trusted boundaries.

Example 2: Restrict NAT Gateway Creation to Network Account

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": ["ec2:CreateNatGateway"],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": ["us-east-1"],
          "aws:PrincipalArn": "arn:aws:iam::111111111111:role/network-admin"
        }
      }
    }
  ]
}

This prevents member accounts from creating their own NAT gateways, forcing them to use the centralized Private NAT Gateway in the network-prod account.

AWS Organizations CLI Setup

#!/bin/bash
# create-aws-organization.sh

# Create organization
aws organizations create-organization --feature-set ALL

# Get root ID dynamically
ROOT_ID=$(aws organizations list-roots --query 'Roots[0].Id' --output text)

# Create organizational units
INFRA_OU=$(aws organizations create-organizational-unit --name Infrastructure_Prod --parent-id $ROOT_ID --query 'OrganizationalUnit.Id' --output text)
PROD_OU=$(aws organizations create-organizational-unit --name Production --parent-id $ROOT_ID --query 'OrganizationalUnit.Id' --output text)
DEV_OU=$(aws organizations create-organizational-unit --name Development --parent-id $ROOT_ID --query 'OrganizationalUnit.Id' --output text)
SANDBOX_OU=$(aws organizations create-organizational-unit --name Sandbox --parent-id $ROOT_ID --query 'OrganizationalUnit.Id' --output text)

# Create accounts and add to OUs
NETWORK_ACCOUNT=$(aws organizations create-account --email "network-prod@example.com" --account-name "network-prod" --query 'CreateAccountStatus.AccountId' --output text)
APP_PROD_ACCOUNT=$(aws organizations create-account --email "app-prod-1@example.com" --account-name "app-prod-1" --query 'CreateAccountStatus.AccountId' --output text)

# Move accounts to appropriate OUs
aws organizations move-account --account-id $NETWORK_ACCOUNT --source-parent-id $ROOT_ID --destination-parent-id $INFRA_OU
aws organizations move-account --account-id $APP_PROD_ACCOUNT --source-parent-id $ROOT_ID --destination-parent-id $PROD_OU

# Enable SCPs
aws organizations enable-policy-type --policy-type SERVICE_CONTROL_POLICY --root-id $ROOT_ID

Component 2: VPC Sharing

What It Is

VPC Sharing (via AWS RAM) allows multiple AWS accounts to create resources in centrally-managed, shared VPC subnets. The VPC owner shares subnets with participant accounts within the same AWS Organization.

How It Works

VPC Owner Account (e.g., network-prod) creates a VPC with subnets
Owner shares subnets via AWS RAM with participant accounts or OUs
Participants can launch EC2, RDS, Lambda, and other resources in shared subnets
Resource Isolation: Participants cannot view, modify, or delete resources belonging to other participants or the owner

Benefits of VPC Sharing

Benefit	Description
Reduced VPC Count	Multiple teams share the same VPC instead of creating separate VPCs
Implicit Routing	Resources in shared subnets communicate via VPC's implicit routing without peering
Simplified Topology	Reduces VPC peering complexity and Transit Gateway attachment count
Separate Billing	Each account maintains independent billing while sharing networking
Access Control	IAM policies control what each participant can do

When to Use VPC Sharing

Use VPC Sharing when:

Teams within the same trust boundary need high interconnectivity
You want to reduce VPC management overhead
Departments share the same trust level (e.g., all production teams)
You want to avoid VPC peering for internal communication

Avoid VPC Sharing when:

Teams require strict network isolation
Different security boundaries exist (e.g., production vs. development)
You need separate NAT gateways per team
Overlapping CIDR blocks are a concern (though Private NAT Gateway solves this)

VPC Sharing Limitations

Participants cannot:

Create, modify, or delete route tables
Create NAT gateways or internet gateways
Modify NACLs
Attach Transit Gateways
Modify shared subnets
Use the default security group (owned by VPC owner)

VPC Sharing Terraform Implementation

# vpc-sharing-owner.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
  # Assume role in network-prod account
  profile = "network-prod-admin"
}

# Create VPC
resource "aws_vpc" "shared_vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "shared-vpc-network-prod"
    Environment = "production"
    OwnedBy     = "network-prod"
  }
}

# Create subnets for sharing
resource "aws_subnet" "shared_subnet_1" {
  vpc_id                  = aws_vpc.shared_vpc.id
  cidr_block              = cidrsubnet(aws_vpc.shared_vpc.cidr_block, 8, 1)
  availability_zone       = data.aws_availability_zones.available.names[0]
  map_public_ip_on_launch = false

  tags = {
    Name        = "shared-subnet-1a"
    SharedWith  = "Production,Development"
  }
}

resource "aws_subnet" "shared_subnet_2" {
  vpc_id                  = aws_vpc.shared_vpc.id
  cidr_block              = cidrsubnet(aws_vpc.shared_vpc.cidr_block, 8, 2)
  availability_zone       = data.aws_availability_zones.available.names[1]
  map_public_ip_on_launch = false

  tags = {
    Name        = "shared-subnet-2b"
    SharedWith  = "Production,Development"
  }
}

data "aws_availability_zones" "available" {
  state = "available"
}

# Get current AWS organizations info
data "aws_organizations_organization" "current" {}

# Get current AWS organizations info
data "aws_organizations_organization" "current" {}

# Enable AWS RAM for organization
resource "aws_ram_resource_share" "subnet_share" {
  name                     = "shared-subnet-resource-share"
  allow_external_principals = false

  tags = {
    Environment = "production"
  }
}

# Associate subnets with resource share
resource "aws_ram_resource_association" "subnet_1_assoc" {
  resource_arn       = aws_subnet.shared_subnet_1.arn
  resource_share_arn = aws_ram_resource_share.subnet_share.arn
}

resource "aws_ram_resource_association" "subnet_2_assoc" {
  resource_arn       = aws_subnet.shared_subnet_2.arn
  resource_share_arn = aws_ram_resource_share.subnet_share.arn
}

# Share with Production OU
resource "aws_ram_principal_association" "production_ou_assoc" {
  principal          = data.aws_organizations_organization.current.id
  resource_share_arn = aws_ram_resource_share.subnet_share.arn
}

VPC Sharing Participant Terraform

# vpc-sharing-participant.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region  = "us-east-1"
  profile = "app-prod-1-admin"  # Participant account
}

# Data source to get shared subnets
data "aws_subnets" "shared" {
  filter {
    name   = "tag:Name"
    values = ["shared-subnet-*"]
  }

  filter {
    name   = "tag:SharedWith"
    values = ["*Production*"]
  }
}

# Data source to get shared VPC
data "aws_vpc" "shared" {
  filter {
    name   = "tag:Name"
    values = ["shared-vpc-network-prod"]
  }
}

# Data source for AMI lookup
data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }
}

# Launch EC2 in shared subnet
resource "aws_instance" "shared_app_instance" {
  ami                    = data.aws_ami.amazon_linux.id
  instance_type          = "t3.medium"
  subnet_id              = data.aws_subnets.shared.ids[0]
  vpc_security_group_ids = [aws_security_group.app_sg.id]

  tags = {
    Name        = "shared-app-instance"
    OwnedBy     = "app-prod-1"
    InSharedVPC = "true"
  }
}

resource "aws_security_group" "app_sg" {
  name        = "app-security-group"
  description = "Security group for app-prod-1"
  vpc_id      = data.aws_vpc.shared.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name     = "app-sg"
    OwnedBy  = "app-prod-1"
  }
}

VPC Sharing CLI Commands

#!/bin/bash
# vpc-sharing-cli.sh

# Create VPC
VPC_ID=$(aws ec2 create-vpc --cidr-block 10.0.0.0/16 --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=shared-vpc}]' --query 'Vpc.VpcId' --output text)

# Get available AZs
AZ1=$(aws ec2 describe-availability-zones --query 'AvailabilityZones[0].ZoneName' --output text)
AZ2=$(aws ec2 describe-availability-zones --query 'AvailabilityZones[1].ZoneName' --output text)

# Create subnets
SUBNET_1=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.1.0/24 --availability-zone $AZ1 --query 'Subnet.SubnetId' --output text)
SUBNET_2=$(aws ec2 create-subnet --vpc-id $VPC_ID --cidr-block 10.0.2.0/24 --availability-zone $AZ2 --query 'Subnet.SubnetId' --output text)

# Get organization ID
ORG_ID=$(aws organizations describe-organization --query 'Organization.Id' --output text)

# Enable AWS RAM for organization
aws ram enable-resource-sharing --organization-id $ORG_ID

# Create resource share
RESOURCE_SHARE=$(aws ram create-resource-share \
  --name "shared-subnet-share" \
  --resource-arns "arn:aws:ec2:us-east-1:$(aws sts get-caller-identity --query Account --output text):subnet/$SUBNET_1" "arn:aws:ec2:us-east-1:$(aws sts get-caller-identity --query Account --output text):subnet/$SUBNET_2" \
  --principals "$ORG_ID" \
  --allow-external-principals false \
  --query 'ResourceShare.ResourceShareArn' --output text)

# Verify in participant account
aws ram get-resource-shares --status ACCEPTED

Component 3: AWS Transit Gateway

What It Is

AWS Transit Gateway is a fully managed, highly available network transit that simplifies VPC connectivity. It acts as a centralized router, eliminating the need for complex VPC peering.

Key Features

Feature	Description
Centralized Routing	Single Transit Gateway replaces N×(N-1)/2 VPC peering connections
Multi-Account Support	Connect VPCs across multiple AWS accounts and regions
Region-Scale	One Transit Gateway per region (supports up to 5,000 VPC attachments)
Peering	Transit Gateway peering for multi-region connectivity
Route Tables	Segregate routes to prevent unwanted communication (e.g., dev → prod)
RAM Sharing	Share Transit Gateway across accounts via AWS RAM

Why Transit Gateway Over VPC Peering

VPC Peering Complexity (N VPCs):
Connections = N × (N-1) / 2
Example: 10 VPCs → 45 peering connections
Example: 50 VPCs → 1,225 peering connections

Transit Gateway Complexity:
Connections = N (one attachment per VPC)
Example: 10 VPCs → 10 attachments
Example: 50 VPCs → 50 attachments

Transit Gateway Architecture

                    ┌─────────────────┐
                    │  network-prod   │
                    │  (Account)      │
                    │                 │
                    │  ┌───────────┐  │
                    │  │ Transit   │  │
                    │  │ Gateway   │  │
                    │  └─────┬─────┘  │
                    └────────┼────────┘
                             │
            ┌────────────────┼────────────────┐
            │                │                │
    ┌───────┴───────┐ ┌──────┴───────┐ ┌─────┴──────┐
    │  app-prod-1   │ │  app-prod-2  │ │ app-dev-1  │
    │  (Account)    │ │  (Account)   │ │ (Account)  │
    │  ┌─────────┐  │ │ ┌─────────┐  │ │ ┌────────┐ │
    │  │ VPC 1   │  │ │ │ VPC 2   │  │ │ │ VPC 3  │ │
    │  └─────────┘  │ │ └─────────┘  │ │ └────────┘ │
    └───────────────┘ └──────────────┘ └────────────┘

Transit Gateway Route Table Segregation

You can create separate route tables within a Transit Gateway to control communication:

Route Table: Production
- app-prod-1 VPC attachment
- app-prod-2 VPC attachment
- network-prod VPC attachment
- NO development routes

Route Table: Development
- app-dev-1 VPC attachment
- app-dev-2 VPC attachment
- network-prod VPC attachment
- NO production routes

This prevents development workloads from accessing production environments through the Transit Gateway.

Transit Gateway with VPC Sharing

When using VPC Sharing:

Only the VPC owner can attach Transit Gateway to shared subnets
Participants cannot attach Transit Gateway
Traffic from participant resources can use Transit Gateway attachments based on routes set by the VPC owner

Transit Gateway Terraform Implementation

# transit-gateway.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region  = "us-east-1"
  profile = "network-prod-admin"
}

# Create Transit Gateway
resource "aws_ec2_transit_gateway" "tgw" {
  description                     = "Enterprise Transit Gateway"
  auto_accept_shared_attachments  = "enable"

  tags = {
    Name        = "enterprise-transit-gateway"
    Environment = "production"
    OwnedBy     = "network-prod"
  }
}

# Create Production Route Table
resource "aws_ec2_transit_gateway_route_table" "prod_rt" {
  transit_gateway_id = aws_ec2_transit_gateway.tgw.id
  tags = {
    Name = "production-route-table"
    OU   = "Production"
  }
}

# Create Development Route Table
resource "aws_ec2_transit_gateway_route_table" "dev_rt" {
  transit_gateway_id = aws_ec2_transit_gateway.tgw.id
  tags = {
    Name = "development-route-table"
    OU   = "Development"
  }
}

# Get current organization info
data "aws_organizations_organization" "current" {}

# Share Transit Gateway via AWS RAM
resource "aws_ram_resource_share" "tgw_share" {
  name                      = "transit-gateway-share"
  allow_external_principals = false

  tags = {
    Environment = "production"
  }
}

resource "aws_ram_resource_association" "tgw_assoc" {
  resource_arn       = aws_ec2_transit_gateway.tgw.arn
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

resource "aws_ram_principal_association" "all_ou_assoc" {
  principal          = data.aws_organizations_organization.current.id
  resource_share_arn = aws_ram_resource_share.tgw_share.arn
}

# Transit Gateway VPC Attachment (owned by network-prod)
resource "aws_ec2_transit_gateway_vpc_attachment" "network_vpc_attach" {
  transit_gateway_id = aws_ec2_transit_gateway.tgw.id
  vpc_id             = aws_vpc.shared_vpc.id
  subnet_ids         = [aws_subnet.shared_subnet_1.id, aws_subnet.shared_subnet_2.id]

  tags = {
    Name = "network-vpc-attach"
  }
}

# Add route to Production Route Table
resource "aws_ec2_transit_gateway_route" "prod_route" {
  destination_cidr_block         = "10.0.0.0/16"
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.network_vpc_attach.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.prod_rt.id
}

Transit Gateway CLI Commands

#!/bin/bash
# transit-gateway-cli.sh

# Get organization ID
ORG_ID=$(aws organizations describe-organization --query 'Organization.Id' --output text)

# Create Transit Gateway
TGW_ID=$(aws ec2 create-transit-gateway \
  --description "Enterprise Transit Gateway" \
  --auto-accept-shared-attachments enable \
  --tag-specifications 'ResourceType=transit-gateway,Tags=[{Key=Name,Value=enterprise-tgw}]' \
  --query 'TransitGateway.TransitGatewayId' --output text)

# Create Production Route Table
PROD_RT=$(aws ec2 create-transit-gateway-route-table \
  --transit-gateway-id $TGW_ID \
  --tag-specifications 'ResourceType=transit-gateway-route-table,Tags=[{Key=Name,Value=prod-rt}]' \
  --query 'TransitGatewayRouteTable.TransitGatewayRouteTableId' --output text)

# Create Development Route Table
DEV_RT=$(aws ec2 create-transit-gateway-route-table \
  --transit-gateway-id $TGW_ID \
  --tag-specifications 'ResourceType=transit-gateway-route-table,Tags=[{Key=Name,Value=dev-rt}]' \
  --query 'TransitGatewayRouteTable.TransitGatewayRouteTableId' --output text)

# Enable RAM for organization
aws ram enable-resource-sharing --organization-id $ORG_ID

# Get current account ID
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Share Transit Gateway
RESOURCE_SHARE=$(aws ram create-resource-share \
  --name "tgw-share" \
  --resource-arns "arn:aws:ec2:us-east-1:$ACCOUNT_ID:transit-gateway/$TGW_ID" \
  --principals "$ORG_ID" \
  --allow-external-principals false \
  --query 'ResourceShare.ResourceShareArn' --output text)

# Create VPC Attachment (requires existing VPC and subnets)
# VPC_ID="vpc-123abc"
# SUBNET_1="subnet-123abc"
# SUBNET_2="subnet-456abc"

# aws ec2 create-transit-gateway-vpc-attachment \
#   --transit-gateway-id $TGW_ID \
#   --vpc-id $VPC_ID \
#   --subnet-ids $SUBNET_1 $SUBNET_2 \
#   --tag-specifications 'ResourceType=transit-gateway-vpc-attachment,Tags=[{Key=Name,Value=network-vpc-attach}]'

# Add route to Production Route Table
# aws ec2 create-transit-gateway-route \
#   --destination-cidr-block 10.0.0.0/16 \
#   --transit-gateway-attachment-id <attachment-id> \
#   --transit-gateway-route-table-id $PROD_RT

Transit Gateway Python Script

#!/usr/env python
# create_transit_gateway.py

import boto3
import json
from typing import Dict, List

class TransitGatewayManager:
    def __init__(self, region: str = "us-east-1", profile: str = "network-prod-admin"):
        self.session = boto3.Session(profile_name=profile, region_name=region)
        self.ec2 = self.session.client("ec2")
        self.ram = self.session.client("ram")

    def create_transit_gateway(self, description: str = "Enterprise Transit Gateway") -> str:
        """Create Transit Gateway with automatic shared attachment acceptance"""
        response = self.ec2.create_transit_gateway(
            Description=description,
            AutoAcceptSharedAttachments="enable",
            Tags=[
                {"Key": "Name", "Value": "enterprise-tgw"},
                {"Key": "Environment", "Value": "production"},
                {"Key": "OwnedBy", "Value": "network-prod"}
            ]
        )
        tgw_id = response["TransitGateway"]["TransitGatewayId"]
        print(f"Created Transit Gateway: {tgw_id}")
        return tgw_id

    def create_route_table(self, tgw_id: str, name: str) -> str:
        """Create Transit Gateway Route Table"""
        response = self.ec2.create_transit_gateway_route_table(
            TransitGatewayId=tgw_id,
            Tags=[{"Key": "Name", "Value": name}]
        )
        rt_id = response["TransitGatewayRouteTable"]["TransitGatewayRouteTableId"]
        print(f"Created Route Table: {rt_id} for {name}")
        return rt_id

    def share_transit_gateway(self, tgw_id: str, organization_id: str = None) -> str:
        """Share Transit Gateway via AWS RAM"""
        # Get organization ID if not provided
        if not organization_id:
            org_response = self.session.client('organizations').describe_organization()
            organization_id = org_response['Organization']['Id']

        # Get current account ID for ARN
        account_id = self.session.client('sts').get_caller_identity()['Account']
        resource_arn = f"arn:aws:ec2:us-east-1:{account_id}:transit-gateway/{tgw_id}"

        response = self.ram.create_resource_share(
            Name="transit-gateway-share",
            ResourceArns=[resource_arn],
            Principals=[organization_id],
            AllowExternalPrincipals=False
        )
        share_arn = response["ResourceShare"]["ResourceShareArn"]
        print(f"Shared Transit Gateway via RAM: {share_arn}")
        return share_arn

    def create_vpc_attachment(
        self, 
        tgw_id: str, 
        vpc_id: str, 
        subnet_ids: List[str],
        name: str = "vpc-attachment"
    ) -> str:
        """Create VPC attachment to Transit Gateway"""
        response = self.ec2.create_transit_gateway_vpc_attachment(
            TransitGatewayId=tgw_id,
            VpcId=vpc_id,
            SubnetIds=subnet_ids,
            Name=name,
            Tags=[
                {"Key": "Name", "Value": name},
                {"Key": "Environment", "Value": "production"}
            ]
        )
        attachment_id = response["TransitGatewayVpcAttachment"]["TransitGatewayAttachmentId"]
        print(f"Created VPC Attachment: {attachment_id}")
        return attachment_id

    def add_route(
        self, 
        rt_id: str, 
        attachment_id: str, 
        cidr: str = "10.0.0.0/16"
    ):
        """Add route to Transit Gateway Route Table"""
        self.ec2.create_transit_gateway_route(
            DestinationCidrBlock=cidr,
            TransitGatewayAttachmentId=attachment_id,
            TransitGatewayRouteTableId=rt_id
        )
        print(f"Added route {cidr} to route table {rt_id}")

    def deploy_enterprise_tgw(self) -> Dict:
        """Deploy complete enterprise Transit Gateway setup"""
        # Create Transit Gateway
        tgw_id = self.create_transit_gateway()

        # Create route tables
        prod_rt = self.create_route_table(tgw_id, "production-route-table")
        dev_rt = self.create_route_table(tgw_id, "development-route-table")

        # Share via RAM (organization ID will be retrieved automatically)
        share_arn = self.share_transit_gateway(tgw_id)

        # Note: VPC attachment requires existing VPC and subnet IDs
        # Uncomment and provide actual IDs when available:
        # vpc_id = "vpc-example-123"
        # subnet_ids = ["subnet-123abc", "subnet-456abc"]
        # attachment_id = self.create_vpc_attachment(tgw_id, vpc_id, subnet_ids)
        # self.add_route(prod_rt, attachment_id)

        return {
            "transit_gateway_id": tgw_id,
            "production_route_table": prod_rt,
            "development_route_table": dev_rt,
            "resource_share_arn": share_arn
            # "vpc_attachment_id": attachment_id  # Uncomment when VPC exists
        }

if __name__ == "__main__":
    manager = TransitGatewayManager()
    result = manager.deploy_enterprise_tgw()
    print(json.dumps(result, indent=2))

Component 4: Private NAT Gateway

What It Is

Private NAT Gateway enables instances in private subnets to connect to other VPCs or on-premises networks through a Transit Gateway, without internet access. Unlike public NAT Gateway, it doesn't use Elastic IPs and doesn't route to internet gateways.

Key Differences: Public vs Private NAT Gateway

Feature	Public NAT Gateway	Private NAT Gateway
Connectivity Type	Public (default)	Private
Elastic IP	Required	Not supported
Internet Access	Yes (via IGW)	No
VPC/On-Prem Access	Yes (via TGW/VGW)	Yes (via TGW/VGW)
Traffic Source IP	Elastic IP	Private NAT Gateway IP
Use Case	Internet egress	Cross-VPC, on-prem connectivity

Why Private NAT Gateway Matters in Multi-Account

Problem 1: Overlapping CIDR Blocks

VPC A (app-dev-1):  100.64.0.0/16  (non-routable, overlapping)
VPC B (app-prod-1): 100.64.0.0/16  (non-routable, overlapping)

Traditional VPC Peering: BLOCKED (overlapping CIDRs)
Transit Gateway: BLOCKED (overlapping CIDRs)
Private NAT Gateway: WORKS (performs source NAT)

Problem 2: Uncontrolled Internet Egress

Each account creating its own NAT Gateway:

50 accounts × 3 AZs = 150 NAT Gateways
150 Elastic IPs @ \$0.10/hr = \$15/hr = \$450/month
No centralized monitoring or filtering

Private NAT Gateway solution:

1 centralized Private NAT Gateway in network-prod
Single Elastic IP (if needed for specific scenarios)
Centralized traffic monitoring and filtering

Private NAT Gateway Architecture

                    ┌─────────────────┐
                    │  network-prod   │
                    │                 │
                    │  ┌───────────┐  │
                    │  │ Private   │  │
                    │  │ NAT GW    │  │
                    │  └─────┬─────┘  │
                    └────────┼────────┘
                             │
                    ┌────────┴────────┐
                    │  Transit Gateway │
                    │                  │
            ┌───────┼───────┐ ┌───────┼───────┐
            │       │       │ │       │       │
    ┌───────┴──┐ ┌──┴──────┐ │ ┌─────┴──┐ ┌──┴──────┐
    │ app-dev-1│ │app-dev-2│ │ │app-prod│ │app-prod-2│
    │ VPC      │ │ VPC     │ │ │ VPC    │ │ VPC      │
    │ 100.64.0 │ │ 100.64.0│ │ │10.1.0  │ │10.2.0    │
    └──────────┘ └─────────┘ │ └────────┘ └──────────┘
                             │
                    ┌────────┴────────┐
                    │ On-Premises     │
                    │ Network         │
                    │ 100.64.0.0/16   │
                    └─────────────────┘

Private NAT Gateway Use Cases

Use Case	Description
Overlapping CIDRs	Connect VPCs with overlapping non-routable CIDRs
On-Premise Approved IPs	Communicate with on-prem networks that only allow specific IPs
Centralized Egress	All accounts use single Private NAT Gateway for cross-VPC/on-prem traffic
Source NAT	Transform source IP from overlapping CIDR to routable Private NAT Gateway IP
Compliance	Meet compliance requirements for approved IP communication only

Private NAT Gateway Implementation Details

VPC A (non-routable):  100.64.0.0/16
  Instance: 100.64.0.10

VPC B (non-routable):  100.64.0.0/16
  ALB: 100.64.0.10 (target)

Solution:
1. Add secondary routable CIDR to VPC A: 10.0.1.0/24
2. Add secondary routable CIDR to VPC B: 10.0.2.0/24
3. Create Private NAT Gateway in VPC A routable subnet with IP: 10.0.1.125
4. Private NAT Gateway performs source NAT: 100.64.0.10 → 10.0.1.125
5. Traffic routes to ALB at 10.0.2.10 (target: 100.64.0.10)
6. Return traffic processed by Private NAT Gateway back to 100.64.0.10

Private NAT Gateway CLI Implementation

#!/bin/bash
# private-nat-gateway-cli.sh

# Get available AZ
AZ=$(aws ec2 describe-availability-zones --query 'AvailabilityZones[0].ZoneName' --output text)

# Create routable subnet for Private NAT Gateway
SUBNET_ID=$(aws ec2 create-subnet \
  --vpc-id vpc-network-prod \
  --cidr-block 10.0.1.0/24 \
  --availability-zone $AZ \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=private-nat-subnet}]' \
  --query 'Subnet.SubnetId' --output text)

# Create Private NAT Gateway
NAT_ID=$(aws ec2 create-nat-gateway \
  --subnet-id $SUBNET_ID \
  --connectivity-type private \
  --tag-specifications 'ResourceType=nat-gateway,Tags=[{Key=Name,Value=private-nat-gateway}]' \
  --query 'NatGateway.NatGatewayId' --output text)

# Wait for Private NAT Gateway to be available
aws ec2 wait nat-gateway-available --nat-gateway-ids $NAT_ID

# Get Private NAT Gateway IP
NAT_IP=$(aws ec2 describe-nat-gateways \
  --nat-gateway-ids $NAT_ID \
  --query 'NatGateways[0].NatGatewayAddresses[0].PrivateIp' --output text)

echo "Private NAT Gateway created: $NAT_ID"
echo "Private NAT Gateway IP: $NAT_IP"

# Create route table for private subnets
RT_ID=$(aws ec2 create-route-table \
  --vpc-id vpc-app-dev-1 \
  --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=private-rt}]' \
  --query 'RouteTable.RouteTableId' --output text)

# Add route to Private NAT Gateway
aws ec2 create-route \
  --route-table-id $RT_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id $NAT_ID

# Associate route table with private subnet (replace with actual subnet ID)
# aws ec2 associate-route-table \
#   --route-table-id $RT_ID \
#   --subnet-id subnet-private-dev-123

Private NAT Gateway Terraform Implementation

# private-nat-gateway.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region  = "us-east-1"
  profile = "network-prod-admin"
}

# Create routable subnet for Private NAT Gateway
resource "aws_subnet" "private_nat_subnet" {
  vpc_id                  = aws_vpc.shared_vpc.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = false

  tags = {
    Name        = "private-nat-subnet"
    ForUse      = "Private NAT Gateway"
  }
}

# Create Private NAT Gateway
resource "aws_nat_gateway" "private_nat" {
  subnet_id         = aws_subnet.private_nat_subnet.id
  connectivity_type = "private"

  tags = {
    Name        = "enterprise-private-nat-gateway"
    Environment = "production"
    OwnedBy     = "network-prod"
    Type        = "private"
  }
}

# Get Private NAT Gateway IP
output "private_nat_ip" {
  value = aws_nat_gateway.private_nat.private_ip
}

output "private_nat_id" {
  value = aws_nat_gateway.private_nat.id
}

# Create route table for private subnets in member accounts
resource "aws_route_table" "private_rt" {
  vpc_id = aws_vpc.shared_vpc.id

  route {
    cidr_block                 = "0.0.0.0/0"
    nat_gateway_id             = aws_nat_gateway.private_nat.id
  }

  tags = {
    Name = "private-route-table"
    ForUse = "Private Subnets"
  }
}

# Associate route table with private subnets
resource "aws_route_table_association" "private_subnet_assoc" {
  subnet_id      = aws_subnet.shared_subnet_1.id
  route_table_id = aws_route_table.private_rt.id
}

Private NAT Gateway Python Script

#!/usr/env python
# create_private_nat_gateway.py

import boto3
import json
from typing import Dict, List

class PrivateNATGatewayManager:
    def __init__(self, region: str = "us-east-1", profile: str = "network-prod-admin"):
        self.session = boto3.Session(profile_name=profile, region_name=region)
        self.ec2 = self.session.client("ec2")

    def create_private_nat_subnet(
        self, 
        vpc_id: str, 
        cidr: str = "10.0.1.0/24",
        az: str = "us-east-1a"
    ) -> str:
        """Create routable subnet for Private NAT Gateway"""
        response = self.ec2.create_subnet(
            VpcId=vpc_id,
            CidrBlock=cidr,
            AvailabilityZone=az,
            Tags=[
                {"Key": "Name", "Value": "private-nat-subnet"},
                {"Key": "ForUse", "Value": "Private NAT Gateway"}
            ]
        )
        subnet_id = response["Subnet"]["SubnetId"]
        print(f"Created Private NAT subnet: {subnet_id}")
        return subnet_id

    def create_private_nat_gateway(self, subnet_id: str, name: str = "private-nat-gateway") -> str:
        """Create Private NAT Gateway with connectivity-type private"""
        response = self.ec2.create_nat_gateway(
            SubnetId=subnet_id,
            ConnectivityType="private",
            Tags=[
                {"Key": "Name", "Value": name},
                {"Key": "Type", "Value": "private"},
                {"Key": "Environment", "Value": "production"}
            ]
        )
        nat_id = response["NatGateway"]["NatGatewayId"]
        print(f"Creating Private NAT Gateway: {nat_id}")

        # Wait for availability
        waiter = self.ec2.get_waiter('nat_gateway_available')
        waiter.wait(NatGatewayIds=[nat_id])

        return nat_id

    def get_private_nat_ip(self, nat_id: str) -> str:
        """Get Private NAT Gateway IP address"""
        response = self.ec2.describe_nat_gateways(NatGatewayIds=[nat_id])
        private_ip = response["NatGateways"][0]["NatGatewayAddresses"][0]["PrivateIp"]
        print(f"Private NAT Gateway IP: {private_ip}")
        return private_ip

    def create_route_table(
        self, 
        vpc_id: str, 
        nat_id: str,
        name: str = "private-rt"
    ) -> str:
        """Create route table with route to Private NAT Gateway"""
        response = self.ec2.create_route_table(
            VpcId=vpc_id,
            Tags=[{"Key": "Name", "Value": name}]
        )
        rt_id = response["RouteTable"]["RouteTableId"]

        # Add route to Private NAT Gateway
        self.ec2.create_route(
            RouteTableId=rt_id,
            DestinationCidrBlock="0.0.0.0/0",
            NatGatewayId=nat_id
        )

        print(f"Created route table: {rt_id} with route to Private NAT Gateway")
        return rt_id

    def associate_route_table(self, rt_id: str, subnet_id: str):
        """Associate route table with subnet"""
        self.ec2.associate_route_table(
            RouteTableId=rt_id,
            SubnetId=subnet_id
        )
        print(f"Associated route table {rt_id} with subnet {subnet_id}")

    def deploy_private_nat(
        self, 
        vpc_id: str,
        subnet_cidr: str = "10.0.1.0/24",
        az: str = "us-east-1a"
    ) -> Dict:
        """Deploy complete Private NAT Gateway setup"""
        # Create subnet
        subnet_id = self.create_private_nat_subnet(vpc_id, subnet_cidr, az)

        # Create Private NAT Gateway
        nat_id = self.create_private_nat_gateway(subnet_id)

        # Get Private IP
        private_ip = self.get_private_nat_ip(nat_id)

        # Create route table
        rt_id = self.create_route_table(vpc_id, nat_id)

        return {
            "nat_gateway_id": nat_id,
            "private_ip": private_ip,
            "subnet_id": subnet_id,
            "route_table_id": rt_id
        }

if __name__ == "__main__":
    manager = PrivateNATGatewayManager()
    result = manager.deploy_private_nat("vpc-network-prod")
    print(json.dumps(result, indent=2))

Complete Enterprise Pattern: Integration

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    AWS ORGANIZATIONS                             │
│  o-1234567890 (Root with All Features + SCPs Enabled)          │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐            │
│  │Infrastructure│ │ Production   │ │ Development  │            │
│  │   (OU)       │ │ (OU)         │ │ (OU)         │            │
│  │              │ │              │ │              │            │
│  │ network-prod │ │ app-prod-1   │ │ app-dev-1    │            │
│  │ (Account)    │ │ (Account)    │ │ (Account)    │            │
│  └──────┬───────┘ └──────┬───────┘ └──────┬───────┘            │
│         │                │                │                     │
└─────────┼────────────────┼────────────────┼─────────────────────┘
          │                │                │
          │    ┌───────────┴───────────┐   │
          │    │   VPC SHARING          │   │
          │    │ (Subnets shared via    │   │
          │    │  AWS RAM)              │   │
          │    └───────────┬───────────┘   │
          │                │                │
          │    ┌───────────┴───────────┐   │
          │    │ Transit Gateway        │   │
          │    │ (Shared via RAM)       │   │
          │    └───────────┬───────────┘   │
          │                │                │
          │    ┌───────────┴───────────┐   │
          │    │ Private NAT Gateway    │   │
          │    │ (Centralized egress)   │   │
          │    └───────────────────────┘   │
          │                                │
┌─────────┼────────────────────────────────┼─────────────────────┐
│         │        Member Accounts         │                     │
│  ┌──────┴───────┐ ┌──────────────────┐ ┌──────────────────┐   │
│  │ app-prod-1   │ │ app-prod-2       │ │ app-dev-1        │   │
│  │ (EC2 in      │ │ (EC2 in          │ │ (EC2 in          │   │
│  │  shared      │ │  shared          │ │  shared          │   │
│  │  subnet)     │ │  subnet)         │ │  subnet)         │   │
│  │ Route via    │ │ Route via        │ │ Route via        │   │
│  │ Private NAT  │ │ Private NAT      │ │ Private NAT      │   │
│  └──────────────┘ └──────────────────┘ └──────────────────┘   │
│                                                               │
└───────────────────────────────────────────────────────────────┘

Data Flow Examples

Example 1: Cross-VPC Communication (Overlapping CIDRs)

1. app-dev-1 (100.64.0.10) sends request to app-prod-1 (100.64.0.20)
2. Traffic routed to Private NAT Gateway (10.0.1.125)
3. Private NAT Gateway performs source NAT: 100.64.0.10 → 10.0.1.125
4. Traffic routed via Transit Gateway to app-prod-1 VPC
5. app-prod-1 receives request from 10.0.1.125 (target: 100.64.0.20)
6. Return traffic processed by Private NAT Gateway back to 100.64.0.10

Example 2: On-Premise Communication (Approved IPs Only)

1. app-dev-1 (10.1.0.10) sends request to on-premise (203.0.113.10)
2. On-premise only allows 10.0.1.125 (Private NAT Gateway IP)
3. Private NAT Gateway performs source NAT: 10.1.0.10 → 10.0.1.125
4. Traffic routed via Transit Gateway → Direct Connect Gateway
5. On-premise receives request from approved IP 10.0.1.125
6. Compliance requirement satisfied

Example 3: Internet Egress (If Needed)

Note: Private NAT Gateway doesn't route to internet
For internet egress, use:
- Public NAT Gateway in network-prod (centralized)
- Or each account's own Public NAT Gateway (if SCP allows)

Why This Pattern Works

Problem	Solution
Overlapping CIDRs	Private NAT Gateway performs source NAT
VPC Peering Complexity	Transit Gateway provides centralized routing
Uncontrolled Egress	Private NAT Gateway centralizes cross-VPC/on-prem traffic
Multiple Accounts	VPC Sharing enables resource isolation with shared networking
Lack of Governance	AWS Organizations + SCPs provide centralized control
Resource Sharing	AWS RAM enables secure cross-account resource sharing

Loses If Not Used

Without This Component	Consequences
AWS Organizations	No centralized governance, inconsistent SCPs, impossible to use RAM
VPC Sharing	50+ VPCs to manage, complex peering, no implicit routing between teams
Transit Gateway	N×(N-1)/2 VPC peering connections, unmanageable at scale
Private NAT Gateway	Overlapping CIDRs blocked, uncontrolled egress, 150+ NAT Gateways
AWS RAM	No secure cross-account resource sharing, manual resource provisioning

Implementation Scripts Summary

1. AWS Organizations Setup

# Create organization with all features
aws organizations create-organization --feature-set ALL

# Enable SCPs (get root-id first)
ROOT_ID=$(aws organizations list-roots --query 'Roots[0].Id' --output text)
aws organizations enable-policy-type --policy-type SERVICE_CONTROL_POLICY --root-id $ROOT_ID

2. SCP Example (Prevent External Sharing)

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": [
      "ram:CreateResourceShare",
      "ram:UpdateResourceShare"
    ],
    "Resource": "*",
    "Condition": {
      "StringNotEquals": {
        "aws:PrincipalOrgID": "o-1234567890"
      }
    }
  }]
}

3. VPC Sharing Terraform

See vpc-sharing-owner.tf and vpc-sharing-participant.tf above

4. Transit Gateway Terraform

See transit-gateway.tf above

5. Private NAT Gateway Terraform

See private-nat-gateway.tf above

6. Python Scripts

create_transit_gateway.py - Deploy enterprise Transit Gateway
create_private_nat_gateway.py - Deploy Private NAT Gateway

FinOps: Cost Optimization and Financial Governance

Overview

FinOps (Financial Operations) is crucial for enterprise multi-account architectures to maintain cost efficiency, visibility, and accountability. This section covers cost optimization strategies, billing transparency, and financial governance for VPC Sharing, Transit Gateway, Private NAT Gateway, and AWS Organizations.

Cost Analysis: Traditional vs. Optimized Architecture

Traditional Multi-Account Networking Costs

Scenario: 50 AWS accounts across 3 environments (Dev, Staging, Prod)

VPC Peering Approach:
├── VPC Peering Connections: 50 × (50-1) ÷ 2 = 1,225 connections
├── Data Transfer: $0.01/GB × 1TB/month × 1,225 = $12,250/month
├── NAT Gateways: 50 accounts × 3 AZs × $45.54/month = $6,831/month
├── Elastic IPs: 150 NAT Gateways × $3.65/month = $547.50/month
├── VPC Management: 50 VPCs × $20/month (operational) = $1,000/month
└── Total Monthly Cost: $20,628.50

Annual Cost: $247,542

Optimized Multi-Account Networking Costs

Optimized Architecture with VPC Sharing + Transit Gateway + Private NAT:

├── Transit Gateway: 1 × $36.50/month = $36.50/month
├── Transit Gateway Attachments: 10 VPCs × $36.50/month = $365/month
├── Data Processing: $0.02/GB × 500GB/month = $10/month
├── Private NAT Gateway: 1 × $45.54/month = $45.54/month
├── Shared VPC Management: 10 VPCs × $20/month = $200/month
├── AWS RAM: No additional cost
└── Total Monthly Cost: $657.04

Annual Cost: $7,884.48
Annual Savings: $239,657.52 (97% reduction)

Component-Level Cost Optimization

1. Transit Gateway Cost Optimization

Pricing Model:

Hourly charge per Transit Gateway: $36.50/month
Hourly charge per attachment: $36.50/month per attachment
Data processing: $0.02 per GB

Cost Optimization Strategies:

# Cost-optimized Transit Gateway configuration
resource "aws_ec2_transit_gateway" "cost_optimized" {
  # Enable automatic route propagation to reduce management overhead
  default_route_table_association = "enable"
  default_route_table_propagation = "enable"

  # Disable DNS support if not needed (reduces data processing)
  dns_support = "disable"

  # Use multicast only if required (additional cost)
  multicast_support = "disable"

  tags = {
    Name = "cost-optimized-tgw"
    CostCenter = "networking"
    Environment = "shared"
  }
}

# Cost allocation tags for attachments
resource "aws_ec2_transit_gateway_vpc_attachment" "cost_tracked" {
  transit_gateway_id = aws_ec2_transit_gateway.cost_optimized.id
  vpc_id            = var.vpc_id
  subnet_ids        = var.subnet_ids

  tags = {
    CostCenter = var.cost_center
    Project = var.project_name
    Environment = var.environment
    Owner = var.team_email
  }
}

FinOps Best Practices:

Strategy	Monthly Savings	Implementation
Consolidate Route Tables	$73/route table	Use shared route tables instead of per-VPC tables
Right-size Attachments	$365/unused attachment	Remove unnecessary VPC attachments
Data Transfer Optimization	$0.02/GB saved	Use VPC endpoints for AWS services
Regional Consolidation	$36.50/region	Single Transit Gateway per region

2. NAT Gateway Cost Optimization

Traditional vs. Private NAT Gateway:

Traditional Approach (50 accounts):
├── NAT Gateways: 50 × 3 AZs × $45.54/month = $6,831/month
├── Data Processing: 150 × $0.045/GB × 100GB = $675/month
├── Elastic IPs: 150 × $3.65/month = $547.50/month
└── Total: $8,053.50/month

Private NAT Gateway Approach:
├── Private NAT Gateway: 1 × $45.54/month = $45.54/month
├── Data Processing: 1 × $0.045/GB × 5000GB = $225/month
├── No Elastic IPs required = $0/month
└── Total: $270.54/month

Monthly Savings: $7,782.96 (96.6% reduction)

Cost-Optimized NAT Gateway Configuration:

resource "aws_nat_gateway" "cost_optimized_private" {
  subnet_id         = aws_subnet.nat_subnet.id
  connectivity_type = "private"

  tags = {
    Name = "cost-optimized-private-nat"
    CostCenter = "networking"
    BillingProject = "shared-infrastructure"
    MonthlyBudget = "500"
  }
}

# Cost monitoring with CloudWatch
resource "aws_cloudwatch_metric_alarm" "nat_cost_alarm" {
  alarm_name          = "nat-gateway-high-cost"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "EstimatedCharges"
  namespace           = "AWS/Billing"
  period              = "86400"
  statistic           = "Maximum"
  threshold           = "300"
  alarm_description   = "This metric monitors NAT Gateway costs"

  dimensions = {
    ServiceName = "AmazonEC2"
    Currency    = "USD"
  }
}

3. VPC Sharing Cost Benefits

Cost Comparison:

Separate VPCs per Account (50 accounts):
├── VPC Endpoints: 50 × 4 endpoints × $7.20/month = $1,440/month
├── Internet Gateways: 50 × $0 (free) = $0/month
├── Route Tables: 50 × 6 tables × $0 = $0/month
├── NACLs: 50 × 5 NACLs × $0 = $0/month
├── Operational Overhead: 50 × $50/month = $2,500/month
└── Total: $3,940/month

Shared VPC Approach (10 shared VPCs):
├── VPC Endpoints: 10 × 4 endpoints × $7.20/month = $288/month
├── Internet Gateways: 10 × $0 = $0/month
├── Route Tables: 10 × 6 tables × $0 = $0/month
├── NACLs: 10 × 5 NACLs × $0 = $0/month
├── Operational Overhead: 10 × $50/month = $500/month
└── Total: $788/month

Monthly Savings: $3,152 (80% reduction)

AWS Organizations FinOps Implementation

1. Consolidated Billing and Cost Allocation

#!/usr/bin/env python
# finops_cost_allocation.py

import boto3
import json
from datetime import datetime, timedelta
from typing import Dict, List

class FinOpsManager:
    def __init__(self):
        self.ce_client = boto3.client('ce')  # Cost Explorer
        self.orgs_client = boto3.client('organizations')
        self.budgets_client = boto3.client('budgets')

    def get_networking_costs_by_account(self, days: int = 30) -> Dict:
        """Get networking costs breakdown by account"""
        end_date = datetime.now().date()
        start_date = end_date - timedelta(days=days)

        response = self.ce_client.get_cost_and_usage(
            TimePeriod={
                'Start': start_date.strftime('%Y-%m-%d'),
                'End': end_date.strftime('%Y-%m-%d')
            },
            Granularity='MONTHLY',
            Metrics=['BlendedCost', 'UnblendedCost'],
            GroupBy=[
                {'Type': 'DIMENSION', 'Key': 'LINKED_ACCOUNT'},
                {'Type': 'DIMENSION', 'Key': 'SERVICE'}
            ],
            Filter={
                'Dimensions': {
                    'Key': 'SERVICE',
                    'Values': [
                        'Amazon Virtual Private Cloud',
                        'Amazon Elastic Compute Cloud - Compute',
                        'AWS Transit Gateway'
                    ]
                }
            }
        )

        cost_breakdown = {}
        for result in response['ResultsByTime']:
            for group in result['Groups']:
                account_id = group['Keys'][0]
                service = group['Keys'][1]
                cost = float(group['Metrics']['BlendedCost']['Amount'])

                if account_id not in cost_breakdown:
                    cost_breakdown[account_id] = {}
                cost_breakdown[account_id][service] = cost

        return cost_breakdown

    def create_networking_budget(self, account_id: str, budget_limit: float):
        """Create budget for networking services per account"""
        budget_name = f"networking-budget-{account_id}"

        budget = {
            'BudgetName': budget_name,
            'BudgetLimit': {
                'Amount': str(budget_limit),
                'Unit': 'USD'
            },
            'TimeUnit': 'MONTHLY',
            'BudgetType': 'COST',
            'CostFilters': {
                'LinkedAccount': [account_id],
                'Service': [
                    'Amazon Virtual Private Cloud',
                    'Amazon Elastic Compute Cloud - Compute',
                    'AWS Transit Gateway'
                ]
            }
        }

        # Create budget with 80% and 100% alerts
        notifications = [
            {
                'Notification': {
                    'NotificationType': 'ACTUAL',
                    'ComparisonOperator': 'GREATER_THAN',
                    'Threshold': 80.0,
                    'ThresholdType': 'PERCENTAGE'
                },
                'Subscribers': [{
                    'SubscriptionType': 'EMAIL',
                    'Address': 'finops-team@company.com'
                }]
            },
            {
                'Notification': {
                    'NotificationType': 'FORECASTED',
                    'ComparisonOperator': 'GREATER_THAN',
                    'Threshold': 100.0,
                    'ThresholdType': 'PERCENTAGE'
                },
                'Subscribers': [{
                    'SubscriptionType': 'EMAIL',
                    'Address': 'finops-team@company.com'
                }]
            }
        ]

        response = self.budgets_client.create_budget(
            AccountId=account_id,
            Budget=budget,
            NotificationsWithSubscribers=notifications
        )

        return response

    def generate_cost_optimization_report(self) -> Dict:
        """Generate comprehensive cost optimization recommendations"""
        recommendations = {
            'vpc_consolidation': self._analyze_vpc_consolidation(),
            'nat_gateway_optimization': self._analyze_nat_gateways(),
            'transit_gateway_efficiency': self._analyze_transit_gateway(),
            'unused_resources': self._find_unused_networking_resources()
        }

        return recommendations

    def _analyze_vpc_consolidation(self) -> Dict:
        """Analyze VPC consolidation opportunities"""
        # Implementation for VPC analysis
        return {
            'current_vpc_count': 50,
            'recommended_vpc_count': 10,
            'potential_monthly_savings': 3152,
            'consolidation_candidates': [
                {'accounts': ['dev-1', 'dev-2', 'dev-3'], 'shared_vpc': 'dev-shared'},
                {'accounts': ['prod-1', 'prod-2'], 'shared_vpc': 'prod-shared'}
            ]
        }

    def _analyze_nat_gateways(self) -> Dict:
        """Analyze NAT Gateway optimization opportunities"""
        return {
            'current_nat_count': 150,
            'recommended_nat_count': 3,
            'potential_monthly_savings': 7782.96,
            'migration_to_private_nat': True
        }

    def _analyze_transit_gateway(self) -> Dict:
        """Analyze Transit Gateway efficiency"""
        return {
            'current_attachments': 50,
            'unused_attachments': 5,
            'potential_monthly_savings': 182.50,
            'route_table_optimization': True
        }

    def _find_unused_networking_resources(self) -> List[Dict]:
        """Find unused networking resources"""
        return [
            {'type': 'Elastic IP', 'count': 10, 'monthly_cost': 36.50},
            {'type': 'NAT Gateway', 'count': 3, 'monthly_cost': 136.62},
            {'type': 'VPC Endpoint', 'count': 5, 'monthly_cost': 36.00}
        ]

if __name__ == "__main__":
    finops = FinOpsManager()

    # Generate cost report
    costs = finops.get_networking_costs_by_account()
    print(json.dumps(costs, indent=2))

    # Generate optimization recommendations
    recommendations = finops.generate_cost_optimization_report()
    print(json.dumps(recommendations, indent=2))

2. Service Control Policies for Cost Governance

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyExpensiveInstanceTypes",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "ec2:InstanceType": [
            "p3.16xlarge",
            "p3.8xlarge",
            "x1e.32xlarge",
            "r5.24xlarge"
          ]
        }
      }
    },
    {
      "Sid": "DenyNATGatewayCreation",
      "Effect": "Deny",
      "Action": "ec2:CreateNatGateway",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalAccount": "111111111111"
        }
      }
    },
    {
      "Sid": "RequireCostCenterTags",
      "Effect": "Deny",
      "Action": [
        "ec2:CreateVpc",
        "ec2:CreateNatGateway",
        "ec2:CreateTransitGateway"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestedRegion": "false",
          "ec2:CreateAction": "false"
        },
        "ForAllValues:StringNotLike": {
          "aws:TagKeys": [
            "CostCenter",
            "Project",
            "Environment"
          ]
        }
      }
    }
  ]
}

Cost Monitoring and Alerting

1. CloudWatch Dashboards for Financial Visibility

resource "aws_cloudwatch_dashboard" "finops_networking" {
  dashboard_name = "FinOps-Networking-Costs"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6

        properties = {
          metrics = [
            ["AWS/Billing", "EstimatedCharges", "ServiceName", "AmazonVPC"],
            ["AWS/Billing", "EstimatedCharges", "ServiceName", "AmazonEC2"],
            ["AWS/Billing", "EstimatedCharges", "ServiceName", "AWSTransitGateway"]
          ]
          view    = "timeSeries"
          stacked = false
          region  = "us-east-1"
          title   = "Networking Service Costs"
          period  = 300
          stat    = "Average"
        }
      },
      {
        type   = "metric"
        x      = 0
        y      = 6
        width  = 6
        height = 6

        properties = {
          metrics = [
            ["AWS/TransitGateway", "BytesIn", "TransitGateway", aws_ec2_transit_gateway.tgw.id],
            ["AWS/TransitGateway", "BytesOut", "TransitGateway", aws_ec2_transit_gateway.tgw.id]
          ]
          view   = "timeSeries"
          region = "us-east-1"
          title  = "Transit Gateway Data Transfer"
          period = 300
        }
      }
    ]
  })
}

2. Automated Cost Optimization

#!/usr/bin/env python
# automated_cost_optimization.py

import boto3
from datetime import datetime, timedelta

class AutomatedCostOptimizer:
    def __init__(self):
        self.ec2 = boto3.client('ec2')
        self.ce = boto3.client('ce')

    def optimize_unused_nat_gateways(self):
        """Identify and recommend deletion of unused NAT Gateways"""
        nat_gateways = self.ec2.describe_nat_gateways()['NatGateways']

        unused_nats = []
        for nat in nat_gateways:
            if nat['State'] == 'available':
                # Check usage in last 7 days
                usage = self._get_nat_gateway_usage(nat['NatGatewayId'])
                if usage < 1000:  # Less than 1GB in 7 days
                    unused_nats.append({
                        'id': nat['NatGatewayId'],
                        'monthly_cost': 45.54,
                        'usage_gb': usage
                    })

        return unused_nats

    def _get_nat_gateway_usage(self, nat_id: str) -> float:
        """Get NAT Gateway usage in GB for last 7 days"""
        # Implementation to get CloudWatch metrics
        # This is a simplified version
        return 0.5  # Placeholder

    def optimize_transit_gateway_attachments(self):
        """Find unused Transit Gateway attachments"""
        attachments = self.ec2.describe_transit_gateway_vpc_attachments()[
            'TransitGatewayVpcAttachments'
        ]

        unused_attachments = []
        for attachment in attachments:
            if attachment['State'] == 'available':
                # Check data transfer in last 30 days
                data_transfer = self._get_attachment_usage(attachment['TransitGatewayAttachmentId'])
                if data_transfer < 100:  # Less than 100MB in 30 days
                    unused_attachments.append({
                        'id': attachment['TransitGatewayAttachmentId'],
                        'vpc_id': attachment['VpcId'],
                        'monthly_cost': 36.50
                    })

        return unused_attachments

    def _get_attachment_usage(self, attachment_id: str) -> float:
        """Get attachment usage in MB for last 30 days"""
        # Implementation to get CloudWatch metrics
        return 50  # Placeholder

    def generate_optimization_report(self) -> dict:
        """Generate comprehensive optimization report"""
        unused_nats = self.optimize_unused_nat_gateways()
        unused_attachments = self.optimize_transit_gateway_attachments()

        total_savings = (
            sum(nat['monthly_cost'] for nat in unused_nats) +
            sum(att['monthly_cost'] for att in unused_attachments)
        )

        return {
            'unused_nat_gateways': unused_nats,
            'unused_tgw_attachments': unused_attachments,
            'total_monthly_savings': total_savings,
            'optimization_actions': [
                f"Delete {len(unused_nats)} unused NAT Gateways",
                f"Remove {len(unused_attachments)} unused TGW attachments"
            ]
        }

if __name__ == "__main__":
    optimizer = AutomatedCostOptimizer()
    report = optimizer.generate_optimization_report()

    print(f"Potential monthly savings: ${report['total_monthly_savings']}")
    for action in report['optimization_actions']:
        print(f"- {action}")

ROI Analysis and Business Case

3-Year Total Cost of Ownership (TCO)

Traditional Multi-Account Networking (3 Years):
├── VPC Peering: $147,000 (data transfer)
├── NAT Gateways: $245,916 (hardware + EIPs)
├── Operational Overhead: $108,000 (management)
├── Scaling Complexity: $50,000 (additional engineering)
└── Total 3-Year TCO: $550,916

Optimized Architecture (3 Years):
├── Transit Gateway: $13,140 (service + attachments)
├── Private NAT Gateway: $1,640 (single gateway)
├── VPC Sharing: $0 (no additional cost)
├── Reduced Operational Overhead: $21,600
└── Total 3-Year TCO: $36,380

Total 3-Year Savings: $514,536 (93.4% reduction)
ROI: 1,414%
Payback Period: 2.1 months

FinOps Governance Framework

Cost Allocation Strategy

Cost Center Allocation Model:

Shared Infrastructure (network-prod account):
├── Transit Gateway: 100% allocated to Infrastructure OU
├── Private NAT Gateway: Allocated based on usage metrics
├── Shared VPC: Split across participant accounts by resource count
└── AWS RAM: No cost allocation needed

Member Account Allocation:
├── EC2 Instances: Direct allocation to account owner
├── Data Transfer: Allocated based on CloudWatch metrics
├── Security Groups: No additional cost
└── Route Table Usage: Included in Transit Gateway allocation

Monthly FinOps Review Process

#!/bin/bash
# monthly_finops_review.sh

# Generate cost reports
aws ce get-cost-and-usage \
  --time-period Start=2024-01-01,End=2024-01-31 \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --output table

# Check budget utilization
aws budgets describe-budgets \
  --account-id $(aws sts get-caller-identity --query Account --output text) \
  --output table

# Generate optimization recommendations
python automated_cost_optimization.py

# Update cost allocation tags
aws resourcegroupstaggingapi get-resources \
  --resource-type-filters "ec2:transit-gateway" \
  --tag-filters Key=CostCenter,Values=networking

Key FinOps Metrics and KPIs

Metric	Target	Current	Trend
Cost per Workload	<$50/month	$15.2/month	↓ 70%
Network Cost % of Total	<15%	8.3%	↓ 45%
Cost Efficiency Ratio	>85%	92.1%	↑ 12%
Resource Utilization	>80%	87.4%	↑ 15%
Cost Avoidance	$20k/month	$28.3k/month	↑ 141%

Best Practices

1. OU-Based SCP Strategy

Infrastructure_Prod OU:
- Allow all networking operations
- Allow RAM sharing within organization
- Allow Transit Gateway attachments

Production OU:
- Deny external RAM sharing
- Deny NAT Gateway creation (use centralized Private NAT)
- Allow Transit Gateway attachments

Development OU:
- Allow NAT Gateway creation (for testing)
- Deny external RAM sharing
- Allow Transit Gateway attachments

Sandbox OU:
- Allow all operations (for experimentation)
- Time-based account expiration

2. Network Account Design

network-prod Account Responsibilities:
- Transit Gateway management
- VPC Sharing (owner)
- Private NAT Gateway
- Public NAT Gateway (if internet egress needed)
- Route 53 Resolver (centralized DNS)
- Network Firewall (centralized filtering)
- IPAM (centralized IP management)

3. Security Considerations

Enable VPC Flow Logs for all shared subnets
Use Security Groups for granular access control
Implement Network ACLs for subnet-level filtering
Use AWS Network Firewall for centralized traffic filtering
Enable CloudTrail for all accounts

4. Cost Optimization

1 Transit Gateway per region (vs. N×(N-1)/2 peering)
1 Private NAT Gateway (vs. 150+ individual NAT Gateways)
Shared VPC reduces VPC count
SCPs prevent accidental expensive operations

Conclusion

This enterprise multi-account networking pattern combining VPC Sharing, Transit Gateway, Private NAT Gateway, and AWS Organizations provides:

Scalability: Supports 50+ VPCs without peering complexity
Security: Isolated workloads with centralized governance
Cost Efficiency: Reduced NAT Gateway and Transit Gateway costs
Flexibility: Handles overlapping CIDRs via Private NAT Gateway
Governance: AWS Organizations + SCPs enforce guardrails
Simplicity: Implicit routing within shared VPC subnets

By implementing this pattern, enterprises can achieve true multi-account architecture benefits while maintaining network connectivity, security, and cost efficiency. The provided CLI, Terraform, and Python scripts enable rapid deployment and customization for your specific requirements.

Remember to:

Start with a well-designed OU structure
Apply SCPs to enforce guardrails
Use AWS RAM for secure resource sharing
Centralize networking in the network-prod account
Monitor costs with AWS Cost Explorer
Continuously review and optimize your architecture

This pattern is production-ready and has been used by enterprises managing hundreds of AWS accounts with complex networking requirements.

Building Highly Available Authentication with Amazon Cognito Multi-Region Replication: Architecture, Migration, and Failover Guide

Manish Kumar — Thu, 04 Jun 2026 15:04:02 +0000

Amazon Cognito's long-awaited Multi-Region Replication (MRR) feature is now generally available, automatically synchronizing user data, credentials, and pool configurations to a secondary AWS Region. Alongside this, AWS has added native support for customer managed KMS keys for encryption control — a critical feature for regulated industries like healthcare and financial services.

Why This Matters

Before MRR, teams building HA authentication on Cognito had to maintain error-prone custom replication solutions using Lambda triggers, DynamoDB Global Tables, and complex sync logic. End users experienced forced password resets during regional failovers, and machine-to-machine (M2M) clients needed to be manually reconfigured in secondary regions.

With MRR, Cognito now:

Automatically replicates user profiles, credentials, MFA secrets, and pool configurations from primary → secondary region
Allows both regions to recognize tokens issued by either region, preserving active sessions
Supports all auth methods — social federation (Google, Apple, Amazon, Facebook), SAML, OIDC, and M2M OAuth2 flows
Provides a built-in Route 53 health check-based failover for custom domains

Architecture Overview

The diagram above shows the complete MRR architecture with:

Prerequisites

Before enabling MRR, your user pool must meet these requirements:

Essentials or Plus feature plan (not available on Lite tier)
Multi-region customer managed KMS key replicated in all target regions
Multi-region OIDC issuer configured on the user pool
A custom domain configured (required for automatic Route 53-based failover)

Step 1: Create a Multi-Region KMS Key

AWS CLI

# Step 1: Create the primary multi-region KMS key in us-west-2
aws kms create-key \
  --region us-west-2 \
  --description "Cognito MRR Key" \
  --multi-region \
  --key-usage ENCRYPT_DECRYPT \
  --origin AWS_KMS \
  --tags TagKey=Purpose,TagValue=CognitoMRR

# Capture the key ARN
PRIMARY_KEY_ARN=$(aws kms list-keys --region us-west-2 \
  --query "Keys[?contains(KeyId, 'mrk')]" \
  --output text | head -1)

# Step 2: Replicate the key to the secondary region
aws kms replicate-key \
  --region us-west-2 \
  --key-id $PRIMARY_KEY_ARN \
  --replica-region us-east-1 \
  --description "Cognito MRR Key Replica (us-east-1)"

# Step 3: Update key policy to allow Cognito access
aws kms put-key-policy \
  --region us-west-2 \
  --key-id $PRIMARY_KEY_ARN \
  --policy-name default \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowCognitoKMSAccess",
        "Effect": "Allow",
        "Principal": {
          "Service": "cognito-idp.amazonaws.com"
        },
        "Action": [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:GenerateDataKey",
          "kms:DescribeKey"
        ],
        "Resource": "*"
      },
      {
        "Sid": "AllowRootAccount",
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:root"
        },
        "Action": "kms:*",
        "Resource": "*"
      }
    ]
  }'

Step 2: Configure the Cognito User Pool

Attach the KMS Key and Configure Multi-Region OIDC Issuer (CLI)

# Update the user pool to use the customer managed KMS key
aws cognito-idp update-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --kms-key-id arn:aws:kms:us-west-2:<ACCOUNT_ID>:key/mrk-XXXXXXXXXX

# Switch the user pool to a multi-region OIDC issuer
# (This is done via the console "Change issuer type" step;
# verify issuer type via describe-user-pool)
aws cognito-idp describe-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --query "UserPool.{IssuerConfiguration:IssuerConfiguration, Domain:Domain}"

⚠️ Important: Switching to a multi-region OIDC issuer changes the iss claim in all tokens. Update all backend services, mobile apps, and SPAs to use the new issuer URL before proceeding.

Step 3: Create the Replica User Pool

AWS CLI

# Create the replica in us-east-1
# Note: The API call is made against the PRIMARY region
aws cognito-idp create-user-pool-replica-region \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --replica-region '{"RegionName": "us-east-1", "KmsKeyId": "arn:aws:kms:us-east-1:<ACCOUNT_ID>:key/mrk-XXXXXXXXXX"}'

# Check replication status — replica info lives on the PRIMARY pool's ReplicaRegions field
aws cognito-idp describe-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --query "UserPool.ReplicaRegions[*].{Region:RegionName, Status:Status}"

# Describe the replica pool directly in the secondary region
aws cognito-idp describe-user-pool \
  --region us-east-1 \
  --user-pool-id us-east-1_XXXXXXXXX \
  --query "UserPool.{Id:Id, Status:Status}"

⚠️ Note: There is no update-user-pool-replica or list-user-pool-replicas CLI command. The replica becomes active automatically once initial sync completes. Replica status is tracked via the primary pool's ReplicaRegions field.

Step 4: Configure Route 53 Health Check & Failover

Normal Traffic Flow

Failover Scenario

CLI Configuration

# Create a Route 53 health check for the primary Cognito endpoint
aws route53 create-health-check \
  --caller-reference "cognito-primary-hc-$(date +%s)" \
  --health-check-config '{
    "Type": "HTTPS",
    "FullyQualifiedDomainName": "cognito-idp.us-west-2.amazonaws.com",
    "Port": 443,
    "RequestInterval": 30,
    "FailureThreshold": 3,
    "ResourcePath": "/health",
    "MeasureLatency": true,
    "Regions": ["us-east-1","us-west-2","eu-west-1"]
  }'

# Store health check ID
HC_ID=$(aws route53 list-health-checks \
  --query "HealthChecks[-1].Id" --output text)

echo "Health Check ID: $HC_ID"

# Update the Cognito custom domain to use this health check for auto-failover
# This is done in the console: Branding > Domain > Edit multi-Region failover
# Associate the $HC_ID with the custom domain

Infrastructure as Code (Terraform)

# main.tf

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.50.0"
    }
  }
}

# ─────────────────────────────────────────────────────────────
# Provider configurations
# ─────────────────────────────────────────────────────────────
provider "aws" {
  alias  = "primary"
  region = "us-west-2"
}

provider "aws" {
  alias  = "secondary"
  region = "us-east-1"
}

data "aws_caller_identity" "current" {}

# ─────────────────────────────────────────────────────────────
# Multi-Region KMS Key
# ─────────────────────────────────────────────────────────────
resource "aws_kms_key" "cognito_mrk" {
  provider                = aws.primary
  description             = "Multi-region KMS key for Cognito MRR"
  multi_region            = true
  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowRoot"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "AllowCognito"
        Effect = "Allow"
        Principal = {
          Service = "cognito-idp.amazonaws.com"
        }
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:GenerateDataKey",
          "kms:DescribeKey",
          "kms:CreateGrant"
        ]
        Resource = "*"
      }
    ]
  })

  tags = {
    Name        = "cognito-mrr-key"
    Environment = "production"
  }
}

resource "aws_kms_alias" "cognito_mrk" {
  provider      = aws.primary
  name          = "alias/cognito-mrr-key"
  target_key_id = aws_kms_key.cognito_mrk.key_id
}

# Replicate the key to secondary region
resource "aws_kms_replica_key" "cognito_mrk_replica" {
  provider                = aws.secondary
  description             = "Replica of Cognito MRR key in us-east-1"
  primary_key_arn         = aws_kms_key.cognito_mrk.arn
  deletion_window_in_days = 30
  enabled                 = true

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowRoot"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "AllowCognito"
        Effect = "Allow"
        Principal = {
          Service = "cognito-idp.amazonaws.com"
        }
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:GenerateDataKey",
          "kms:DescribeKey",
          "kms:CreateGrant"
        ]
        Resource = "*"
      }
    ]
  })

  tags = {
    Name        = "cognito-mrr-key-replica"
    Environment = "production"
  }
}

# ─────────────────────────────────────────────────────────────
# Primary Cognito User Pool
# ─────────────────────────────────────────────────────────────
resource "aws_cognito_user_pool" "primary" {
  provider = aws.primary
  name     = "myapp-user-pool-primary"

  # Use customer managed KMS key
  user_pool_add_ons {
    advanced_security_mode = "ENFORCED"
  }

  password_policy {
    minimum_length                   = 12
    require_lowercase                = true
    require_uppercase                = true
    require_numbers                  = true
    require_symbols                  = true
    temporary_password_validity_days = 7
  }

  mfa_configuration = "OPTIONAL"

  software_token_mfa_configuration {
    enabled = true
  }

  # Email verification
  auto_verified_attributes = ["email"]

  account_recovery_setting {
    recovery_mechanism {
      name     = "verified_email"
      priority = 1
    }
  }

  schema {
    name                     = "email"
    attribute_data_type      = "String"
    required                 = true
    mutable                  = true
  }

  tags = {
    Name        = "myapp-primary"
    Environment = "production"
    Region      = "us-west-2"
  }
}

# App Client for the primary pool
resource "aws_cognito_user_pool_client" "primary" {
  provider        = aws.primary
  name            = "myapp-client-primary"
  user_pool_id    = aws_cognito_user_pool.primary.id

  explicit_auth_flows = [
    "ALLOW_USER_SRP_AUTH",
    "ALLOW_REFRESH_TOKEN_AUTH",
    "ALLOW_USER_PASSWORD_AUTH"
  ]

  access_token_validity  = 60
  id_token_validity      = 60
  refresh_token_validity = 30

  token_validity_units {
    access_token  = "minutes"
    id_token      = "minutes"
    refresh_token = "days"
  }

  prevent_user_existence_errors = "ENABLED"
}

# ─────────────────────────────────────────────────────────────
# Replica User Pool (Secondary Region)
# NOTE: There is no standalone aws_cognito_user_pool_replica resource in
# the AWS Terraform provider. Replication is configured via the
# replica_regions block inside aws_cognito_user_pool.
# ─────────────────────────────────────────────────────────────
# Add a replica_regions block to aws_cognito_user_pool.primary:
#
# resource "aws_cognito_user_pool" "primary" {
#   ...
#   replica_regions {
#     region_name = "us-east-1"
#     kms_key_id  = aws_kms_replica_key.cognito_mrk_replica.arn
#   }
# }
#
# Reference: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cognito_user_pool

# ─────────────────────────────────────────────────────────────
# Route 53 Health Check for Failover
# ─────────────────────────────────────────────────────────────
resource "aws_route53_health_check" "cognito_primary" {
  fqdn              = "cognito-idp.us-west-2.amazonaws.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30

  tags = {
    Name = "cognito-primary-health-check"
  }
}

# ─────────────────────────────────────────────────────────────
# CloudWatch Alarm for Failover Monitoring
# ─────────────────────────────────────────────────────────────
resource "aws_cloudwatch_metric_alarm" "cognito_errors_primary" {
  provider            = aws.primary
  alarm_name          = "cognito-high-error-rate-primary"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "Errors"
  namespace           = "AWS/Cognito"
  period              = 60
  statistic           = "Sum"
  threshold           = 10

  dimensions = {
    UserPool       = aws_cognito_user_pool.primary.id
    UserPoolClient = aws_cognito_user_pool_client.primary.id
  }

  alarm_description = "Cognito primary region error rate too high - consider failover"
  alarm_actions     = [aws_sns_topic.cognito_alerts.arn]
}

resource "aws_sns_topic" "cognito_alerts" {
  provider = aws.primary
  name     = "cognito-mrr-alerts"
}

# ─────────────────────────────────────────────────────────────
# Outputs
# ─────────────────────────────────────────────────────────────
output "primary_user_pool_id" {
  value = aws_cognito_user_pool.primary.id
}

output "primary_user_pool_endpoint" {
  value = aws_cognito_user_pool.primary.endpoint
}

# Replica pool ID is obtained from describe-user-pool in the secondary region,
# not from a separate Terraform resource output.

output "kms_primary_key_arn" {
  value = aws_kms_key.cognito_mrk.arn
}

output "kms_replica_key_arn" {
  value = aws_kms_replica_key.cognito_mrk_replica.arn
}

output "route53_health_check_id" {
  value = aws_route53_health_check.cognito_primary.id
}

Python Automation Scripts

Script 1: Full Setup Orchestrator

#!/usr/bin/env python3
"""
cognito_mrr_setup.py
Automates Amazon Cognito Multi-Region Replication setup using boto3.
"""

import boto3
import json
import time
import logging
from typing import Optional

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s"
)
log = logging.getLogger(__name__)

PRIMARY_REGION   = "us-west-2"
SECONDARY_REGION = "us-east-1"
ACCOUNT_ID       = boto3.client("sts").get_caller_identity()["Account"]


# ─────────────────────────────────────────────────────────────
# KMS: Create and Replicate a Multi-Region Key
# ─────────────────────────────────────────────────────────────
def create_multi_region_kms_key(primary_region: str) -> str:
    kms = boto3.client("kms", region_name=primary_region)

    key_policy = json.dumps({
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowRoot",
                "Effect": "Allow",
                "Principal": {"AWS": f"arn:aws:iam::{ACCOUNT_ID}:root"},
                "Action": "kms:*",
                "Resource": "*"
            },
            {
                "Sid": "AllowCognito",
                "Effect": "Allow",
                "Principal": {"Service": "cognito-idp.amazonaws.com"},
                "Action": [
                    "kms:Encrypt", "kms:Decrypt",
                    "kms:GenerateDataKey", "kms:DescribeKey", "kms:CreateGrant"
                ],
                "Resource": "*"
            }
        ]
    })

    response = kms.create_key(
        Description="Multi-region KMS key for Cognito MRR",
        MultiRegion=True,
        KeyUsage="ENCRYPT_DECRYPT",
        Origin="AWS_KMS",
        Policy=key_policy,
        Tags=[{"TagKey": "Purpose", "TagValue": "CognitoMRR"}]
    )

    key_arn = response["KeyMetadata"]["Arn"]
    key_id  = response["KeyMetadata"]["KeyId"]
    log.info(f"✅ Created multi-region KMS key: {key_arn}")

    kms.create_alias(AliasName="alias/cognito-mrr-key", TargetKeyId=key_id)
    return key_arn


def replicate_kms_key(primary_key_arn: str, target_region: str) -> str:
    kms = boto3.client("kms", region_name=PRIMARY_REGION)

    response = kms.replicate_key(
        KeyId=primary_key_arn,
        ReplicaRegion=target_region,
        Description=f"Cognito MRR key replica in {target_region}"
    )

    replica_arn = response["ReplicaKeyMetadata"]["Arn"]
    log.info(f"✅ Replicated KMS key to {target_region}: {replica_arn}")

    # Update replica key policy for Cognito access
    kms_secondary = boto3.client("kms", region_name=target_region)
    kms_secondary.put_key_policy(
        KeyId=replica_arn,
        PolicyName="default",
        Policy=json.dumps({
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "AllowRoot",
                    "Effect": "Allow",
                    "Principal": {"AWS": f"arn:aws:iam::{ACCOUNT_ID}:root"},
                    "Action": "kms:*",
                    "Resource": "*"
                },
                {
                    "Sid": "AllowCognito",
                    "Effect": "Allow",
                    "Principal": {"Service": "cognito-idp.amazonaws.com"},
                    "Action": [
                        "kms:Encrypt", "kms:Decrypt",
                        "kms:GenerateDataKey", "kms:DescribeKey", "kms:CreateGrant"
                    ],
                    "Resource": "*"
                }
            ]
        })
    )
    return replica_arn


# ─────────────────────────────────────────────────────────────
# Cognito: Create User Pool with KMS Encryption
# ─────────────────────────────────────────────────────────────
def create_primary_user_pool(kms_key_arn: str, region: str) -> str:
    cognito = boto3.client("cognito-idp", region_name=region)

    response = cognito.create_user_pool(
        PoolName="myapp-user-pool-primary",
        Policies={
            "PasswordPolicy": {
                "MinimumLength": 12,
                "RequireUppercase": True,
                "RequireLowercase": True,
                "RequireNumbers": True,
                "RequireSymbols": True,
                "TemporaryPasswordValidityDays": 7
            }
        },
        MfaConfiguration="OPTIONAL",
        UserPoolAddOns={"AdvancedSecurityMode": "ENFORCED"},
        AutoVerifiedAttributes=["email"],
        Schema=[
            {
                "Name": "email",
                "AttributeDataType": "String",
                "Required": True,
                "Mutable": True
            }
        ],
        UserPoolTags={
            "Environment": "production",
            "Region": region,
            "MRR": "enabled"
        },
        UserPoolEncryptionConfig={"KMSKeyID": kms_key_arn}
    )

    pool_id = response["UserPool"]["Id"]
    log.info(f"✅ Created primary user pool: {pool_id}")
    return pool_id


# ─────────────────────────────────────────────────────────────
# Cognito: Create Replica User Pool
# ─────────────────────────────────────────────────────────────
def create_user_pool_replica(
    primary_pool_id: str,
    target_region: str,
    source_region: str = PRIMARY_REGION
) -> dict:
    cognito = boto3.client("cognito-idp", region_name=source_region)

    response = cognito.create_user_pool_replica_region(
        UserPoolId=primary_pool_id,
        ReplicaRegion={
            "RegionName": target_region
        }
    )

    replica = response["UserPoolReplica"]
    log.info(
        f"✅ Created replica user pool in {target_region}\n"
        f"   ARN: {replica['UserPoolArn']}\n"
        f"   Status: {replica['Status']}"
    )
    return replica


# ─────────────────────────────────────────────────────────────
# Cognito: Poll until replica is INACTIVE (synced), then ACTIVATE
# ─────────────────────────────────────────────────────────────
def wait_and_activate_replica(
    replica_pool_id: str,
    secondary_region: str,
    timeout_seconds: int = 600
):
    cognito = boto3.client("cognito-idp", region_name=secondary_region)
    elapsed = 0

    log.info(f"⏳ Waiting for replica pool {replica_pool_id} to be ready...")

    while elapsed < timeout_seconds:
        resp = cognito.describe_user_pool(UserPoolId=replica_pool_id)
        status = resp["UserPool"].get("Status", "UNKNOWN")
        log.info(f"   Pool status: {status} ({elapsed}s elapsed)")

        if status == "ACTIVE":
            # Replica becomes ACTIVE automatically once initial sync completes
            log.info(f"✅ Replica pool {replica_pool_id} is ACTIVE")
            return

        time.sleep(30)
        elapsed += 30

    raise TimeoutError(f"Replica pool did not become INACTIVE within {timeout_seconds}s")


# ─────────────────────────────────────────────────────────────
# Route 53: Create Health Check
# ─────────────────────────────────────────────────────────────
def create_route53_health_check(primary_region: str) -> str:
    r53 = boto3.client("route53")

    response = r53.create_health_check(
        CallerReference=f"cognito-hc-{int(time.time())}",
        HealthCheckConfig={
            "Type": "HTTPS",
            "FullyQualifiedDomainName": f"cognito-idp.{primary_region}.amazonaws.com",
            "Port": 443,
            "RequestInterval": 30,
            "FailureThreshold": 3,
            "MeasureLatency": True,
            "Regions": ["us-east-1", "us-west-2", "eu-west-1"]
        }
    )

    hc_id = response["HealthCheck"]["Id"]
    log.info(f"✅ Created Route 53 health check: {hc_id}")

    r53.change_tags_for_resource(
        ResourceType="healthcheck",
        ResourceId=hc_id,
        AddTags=[{"Key": "Name", "Value": "cognito-primary-hc"}]
    )
    return hc_id


# ─────────────────────────────────────────────────────────────
# CloudWatch: Alarms and SNS notifications
# ─────────────────────────────────────────────────────────────
def setup_monitoring(
    pool_id: str,
    client_id: str,
    region: str,
    alert_email: Optional[str] = None
) -> str:
    sns = boto3.client("sns", region_name=region)
    cw  = boto3.client("cloudwatch", region_name=region)

    # Create SNS topic
    topic = sns.create_topic(Name="cognito-mrr-alerts")
    topic_arn = topic["TopicArn"]

    if alert_email:
        sns.subscribe(
            TopicArn=topic_arn,
            Protocol="email",
            Endpoint=alert_email
        )
        log.info(f"📧 Subscribed {alert_email} to alerts topic")

    # Create CloudWatch alarm for auth errors
    cw.put_metric_alarm(
        AlarmName="cognito-primary-high-errors",
        AlarmDescription="Cognito primary region auth error rate is high - consider failover",
        MetricName="Errors",
        Namespace="AWS/Cognito",
        Dimensions=[
            {"Name": "UserPool", "Value": pool_id},
            {"Name": "UserPoolClient", "Value": client_id}
        ],
        Statistic="Sum",
        Period=60,
        EvaluationPeriods=3,
        Threshold=10,
        ComparisonOperator="GreaterThanThreshold",
        AlarmActions=[topic_arn],
        OKActions=[topic_arn],
        TreatMissingData="notBreaching"
    )

    # Alarm for sign-in latency
    cw.put_metric_alarm(
        AlarmName="cognito-primary-high-latency",
        AlarmDescription="Cognito primary region sign-in latency > 2000ms",
        MetricName="SignInSuccesses",
        Namespace="AWS/Cognito",
        Dimensions=[{"Name": "UserPool", "Value": pool_id}],
        Statistic="p99",
        Period=60,
        EvaluationPeriods=5,
        Threshold=2000,
        ComparisonOperator="GreaterThanThreshold",
        AlarmActions=[topic_arn],
        TreatMissingData="notBreaching"
    )

    log.info("✅ CloudWatch monitoring configured")
    return topic_arn


# ─────────────────────────────────────────────────────────────
# Main Orchestration
# ─────────────────────────────────────────────────────────────
def main():
    log.info("🚀 Starting Cognito Multi-Region Replication setup...")

    # 1. Create multi-region KMS key
    primary_key_arn = create_multi_region_kms_key(PRIMARY_REGION)

    # 2. Replicate KMS key to secondary region
    replica_key_arn = replicate_kms_key(primary_key_arn, SECONDARY_REGION)

    # 3. Create primary Cognito user pool with KMS encryption
    primary_pool_id = create_primary_user_pool(primary_key_arn, PRIMARY_REGION)

    # NOTE: Before calling create_user_pool_replica, you must:
    # a) Update the user pool to use the multi-region OIDC issuer (via console)
    # b) Update your applications with the new OIDC issuer URLs

    # 4. Create replica user pool
    replica = create_user_pool_replica(primary_pool_id, SECONDARY_REGION)
    # Extract the replica pool ID from the ARN
    replica_pool_id = replica["UserPoolArn"].split("/")[-1]

    # 5. Wait for replication to complete and activate
    wait_and_activate_replica(replica_pool_id, SECONDARY_REGION)

    # 6. Set up Route 53 health check
    health_check_id = create_route53_health_check(PRIMARY_REGION)

    # 7. Set up monitoring and alerting
    cognito_primary = boto3.client("cognito-idp", region_name=PRIMARY_REGION)
    clients = cognito_primary.list_user_pool_clients(UserPoolId=primary_pool_id)
    client_id = clients["UserPoolClients"][0]["ClientId"] if clients["UserPoolClients"] else "NONE"

    setup_monitoring(
        pool_id=primary_pool_id,
        client_id=client_id,
        region=PRIMARY_REGION,
        alert_email=os.environ.get("ALERT_EMAIL", "")
    )

    log.info("\n" + "="*60)
    log.info("✅ COGNITO MULTI-REGION REPLICATION SETUP COMPLETE")
    log.info("="*60)
    log.info(f"Primary Pool ID   : {primary_pool_id}")
    log.info(f"Primary Region    : {PRIMARY_REGION}")
    log.info(f"Replica Pool ID   : {replica_pool_id}")
    log.info(f"Secondary Region  : {SECONDARY_REGION}")
    log.info(f"KMS Key (Primary) : {primary_key_arn}")
    log.info(f"KMS Key (Replica) : {replica_key_arn}")
    log.info(f"Route53 HC ID     : {health_check_id}")
    log.info("="*60)


if __name__ == "__main__":
    main()

Script 2: Failover Health Monitor (Lambda-Compatible)

#!/usr/bin/env python3
"""
cognito_failover_monitor.py
Monitors primary Cognito health and can be deployed as a Lambda function
to automate failover decisions or send alerts.
"""

import boto3
import json
import logging
import os
from datetime import datetime, timezone, timedelta

log = logging.getLogger()
log.setLevel(logging.INFO)

PRIMARY_REGION   = os.environ.get("PRIMARY_REGION", "us-west-2")
SECONDARY_REGION = os.environ.get("SECONDARY_REGION", "us-east-1")
PRIMARY_POOL_ID  = os.environ.get("PRIMARY_POOL_ID", "")
REPLICA_POOL_ID  = os.environ.get("REPLICA_POOL_ID", "")
SNS_TOPIC_ARN    = os.environ.get("SNS_TOPIC_ARN", "")
HC_ID            = os.environ.get("ROUTE53_HEALTH_CHECK_ID", "")


def get_cognito_error_rate(pool_id: str, region: str) -> float:
    """Returns the error count in the last 5 minutes."""
    cw = boto3.client("cloudwatch", region_name=region)
    end   = datetime.now(timezone.utc)
    start = end - timedelta(minutes=5)

    resp = cw.get_metric_statistics(
        Namespace="AWS/Cognito",
        MetricName="Errors",
        Dimensions=[{"Name": "UserPool", "Value": pool_id}],
        StartTime=start,
        EndTime=end,
        Period=300,
        Statistics=["Sum"]
    )

    datapoints = resp.get("Datapoints", [])
    return datapoints[0]["Sum"] if datapoints else 0.0


def get_route53_health_check_status(hc_id: str) -> str:
    r53 = boto3.client("route53")
    resp = r53.get_health_check_status(HealthCheckId=hc_id)
    statuses = resp.get("HealthCheckObservations", [])
    healthy = sum(1 for s in statuses if s["StatusReport"]["Status"].startswith("Success"))
    total   = len(statuses)
    return "HEALTHY" if healthy > (total / 2) else "UNHEALTHY"


def describe_replica_status(pool_id: str, region: str) -> str:
    cognito = boto3.client("cognito-idp", region_name=region)
    resp    = cognito.describe_user_pool(UserPoolId=pool_id)
    return resp["UserPool"].get("Status", "UNKNOWN")


def send_alert(message: str, subject: str):
    if not SNS_TOPIC_ARN:
        log.warning("SNS_TOPIC_ARN not set — skipping alert")
        return
    sns = boto3.client("sns")
    sns.publish(TopicArn=SNS_TOPIC_ARN, Message=message, Subject=subject)
    log.info(f"📢 Alert sent: {subject}")


def lambda_handler(event, context):
    """
    Lambda entry point.
    Checks Cognito primary health, logs status, and sends alert if degraded.
    """
    log.info("🔍 Running Cognito MRR health check...")

    error_count   = get_cognito_error_rate(PRIMARY_POOL_ID, PRIMARY_REGION)
    hc_status     = get_route53_health_check_status(HC_ID) if HC_ID else "NOT_CONFIGURED"
    replica_status = describe_replica_status(REPLICA_POOL_ID, SECONDARY_REGION) if REPLICA_POOL_ID else "N/A"

    report = {
        "timestamp":      datetime.now(timezone.utc).isoformat(),
        "primary_region": PRIMARY_REGION,
        "error_count_5m": error_count,
        "route53_hc":     hc_status,
        "replica_status": replica_status,
        "recommendation": "FAILOVER" if (error_count > 10 or hc_status == "UNHEALTHY") else "HEALTHY"
    }

    log.info(json.dumps(report, indent=2))

    if report["recommendation"] == "FAILOVER":
        send_alert(
            message=json.dumps(report, indent=2),
            subject="⚠️ Cognito Primary Region Degraded — Consider Failover"
        )

    return {"statusCode": 200, "body": json.dumps(report)}


# Local testing entry point
if __name__ == "__main__":
    result = lambda_handler({}, {})
    print(result)

Key Limitations to Know

Limitation	Details
Write operations	Secondary pools are read-only — no new sign-ups, password resets, or profile edits during failover
TOTP MFA	Not supported in secondary replicas; TOTP users must authenticate via primary
Replica count	Maximum one secondary replica per user pool
Federated users	Must have previously signed in via primary before they can use the replica
Lockout counts	Failed auth attempt counters are not synced across regions
Custom domain required	Automatic Route 53 failover only works with a custom domain
Feature plan	Requires Essentials or Plus tier — not available on Lite

Pricing Summary

Auth Type	Essentials Tier	Plus Tier
User Authentication	\$0.0045 / MAU / replica region	\$0.006 / MAU / replica region
M2M Authentication	+30% on standard token pricing	+30% on standard token pricing

Pricing details are per replica region added on top of standard Cognito costs.

Available Regions

MRR is available across major AWS regions as of June 2026, including US East/West, EU (Frankfurt, Ireland, London, Paris, Stockholm), APAC (Mumbai, Tokyo, Seoul, Singapore, Sydney), Canada (Central), and South America (São Paulo). Any of these can serve as either the source or destination for replication.

Operational Checklist Before Going Live

[ ] Upgrade user pool to Essentials or Plus plan
[ ] Create a multi-region KMS key and replicate it to target region
[ ] Update key policy to allow cognito-idp.amazonaws.com access
[ ] Switch to multi-region OIDC issuer and update all app clients
[ ] Deploy Lambda triggers, WAF rules, and logging config in secondary region
[ ] Create replica and wait for INACTIVE → ACTIVE transition
[ ] Set up Route 53 health check and link it to your custom domain's failover config
[ ] Configure CloudWatch alarms for error rates and latency
[ ] Test failover during off-peak hours by routing a small traffic slice to secondary
[ ] Disable sign-up/password-reset UI elements when operating in failover mode

Migrating Existing Cognito User Pools to Multi-Region Replication

Migrating an existing Cognito user pool to MRR is more involved than a fresh setup because you have live users, active sessions, and applications already hardcoded to the original OIDC issuer URL. This guide walks you through every phase — eligibility check, issuer migration, KMS attachment, replica creation, and app updates — without forcing users to re-authenticate or reset passwords.

Phase 0: Eligibility Check — Are You on Next-Gen Infrastructure?

This is the most critical gating factor. MRR only works on next-generation Cognito infrastructure. Older existing pools will be automatically upgraded by AWS, but they cannot self-opt-in. Until then, the console shows an exception message on ineligible pools.

Check Your Pool's Eligibility via CLI

# Check your user pool details for infrastructure version
aws cognito-idp describe-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --query "UserPool.{Tier:UserPoolTier, Status:Status, Domain:Domain}"

# Check if MRR options are available by inspecting ReplicaRegions on describe-user-pool
aws cognito-idp describe-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --query "UserPool.{Tier:UserPoolTier, ReplicaRegions:ReplicaRegions}"

⚠️ Note: There is no list-user-pool-replicas CLI command. Replica information is returned via the ReplicaRegions field in describe-user-pool on the primary pool. If the field is absent or the feature returns an error, the pool is not yet on next-gen infrastructure.

💡 Tip: Check the AWS Security Blog post on Cognito next-generation infrastructure to understand the upgrade timeline.

Phase 1: Pre-Migration Audit

Before touching anything, run a full audit of your existing pool. This prevents surprises mid-migration.

Python: Audit Script for Existing Pool

#!/usr/bin/env python3
"""
cognito_mrr_audit.py
Audits an existing Cognito user pool for MRR readiness.
Outputs a checklist of items that need remediation.
"""

import boto3
import json
import sys
from dataclasses import dataclass, field
from typing import List

@dataclass
class AuditResult:
    check: str
    status: str       # PASS / FAIL / WARN / INFO
    detail: str
    action_required: str = ""

def audit_pool_for_mrr(pool_id: str, region: str) -> List[AuditResult]:
    cognito = boto3.client("cognito-idp", region_name=region)
    results = []

    pool = cognito.describe_user_pool(UserPoolId=pool_id)["UserPool"]

    # ── 1. Feature Plan (Tier) ────────────────────────────────
    tier = pool.get("UserPoolTier", "LITE")
    results.append(AuditResult(
        check="Feature Plan",
        status="PASS" if tier in ["ESSENTIALS", "PLUS"] else "FAIL",
        detail=f"Current tier: {tier}",
        action_required="" if tier != "LITE" else "Upgrade to Essentials or Plus tier before enabling MRR"
    ))

    # ── 2. KMS Key Configuration ──────────────────────────────
    kms_config = pool.get("UserPoolEncryptionConfig", {})
    kms_key_id = kms_config.get("KMSKeyID", "")
    if kms_key_id:
        kms = boto3.client("kms", region_name=region)
        key_meta = kms.describe_key(KeyId=kms_key_id)["KeyMetadata"]
        is_mrk = key_meta.get("MultiRegion", False)
        results.append(AuditResult(
            check="KMS Key",
            status="PASS" if is_mrk else "FAIL",
            detail=f"Key ARN: {kms_key_id}, MultiRegion: {is_mrk}",
            action_required="" if is_mrk else "Replace with a multi-region KMS key (mrk- prefix)"
        ))
    else:
        results.append(AuditResult(
            check="KMS Key",
            status="FAIL",
            detail="No customer managed KMS key configured",
            action_required="Create a multi-region KMS key and attach it to the user pool"
        ))

    # ── 3. OIDC Issuer Type ───────────────────────────────────
    issuer_config = pool.get("IssuerConfiguration", {})
    issuer_type   = issuer_config.get("Type", "LEGACY")
    results.append(AuditResult(
        check="OIDC Issuer Type",
        status="PASS" if issuer_type == "UPDATED" else "FAIL",
        detail=f"Issuer type: {issuer_type}",
        action_required="" if issuer_type == "UPDATED" else (
            "Switch to UPDATED issuer — WARNING: breaking change for existing apps. "
            "Update all services that validate the 'iss' claim before switching."
        )
    ))

    # ── 4. Current Issuer URL ─────────────────────────────────
    old_issuer = f"https://cognito-idp.{region}.amazonaws.com/{pool_id}"
    new_issuer = f"https://issuer-cognito-idp.{region}.amazonaws.com/{pool_id}"
    results.append(AuditResult(
        check="Issuer URL (for reference)",
        status="INFO",
        detail=f"OLD: {old_issuer}\nNEW: {new_issuer}",
        action_required="Update all apps, API GWs, and JWK validators to use NEW issuer URL"
    ))

    # ── 5. MFA Configuration ──────────────────────────────────
    mfa = pool.get("MfaConfiguration", "OFF")
    totp_enabled = pool.get("UserPoolAddOns", {}).get("AdvancedSecurityMode", "OFF")
    results.append(AuditResult(
        check="TOTP MFA",
        status="WARN" if mfa != "OFF" else "PASS",
        detail=f"MFA Config: {mfa}",
        action_required="TOTP MFA users CANNOT authenticate on the replica. "
                        "Plan a communication strategy and disable TOTP-reliant flows in failover mode."
                        if mfa != "OFF" else ""
    ))

    # ── 6. Custom Domain ──────────────────────────────────────
    domain = pool.get("Domain", "") or pool.get("CustomDomain", "")
    results.append(AuditResult(
        check="Custom Domain",
        status="PASS" if domain else "WARN",
        detail=f"Domain: {domain or 'NOT CONFIGURED'}",
        action_required="Automatic Route 53 failover requires a custom domain. "
                        "Without it, your app must manually switch regional endpoints."
                        if not domain else ""
    ))

    # ── 7. App Clients ────────────────────────────────────────
    clients = cognito.list_user_pool_clients(UserPoolId=pool_id, MaxResults=10)
    client_count = len(clients["UserPoolClients"])
    results.append(AuditResult(
        check="App Clients",
        status="INFO",
        detail=f"{client_count} app client(s) found — will be auto-replicated after MRR enabled",
        action_required="Verify each client's callback URLs and allowed OAuth flows post-migration"
    ))

    # ── 8. Lambda Triggers ────────────────────────────────────
    triggers = pool.get("LambdaConfig", {})
    has_triggers = bool(triggers)
    results.append(AuditResult(
        check="Lambda Triggers",
        status="WARN" if has_triggers else "PASS",
        detail=f"Triggers configured: {list(triggers.keys()) if has_triggers else 'None'}",
        action_required="Lambda triggers must be separately configured for the replica region. "
                        "Cross-region Lambda invocations won't work automatically."
                        if has_triggers else ""
    ))

    # ── 9. User Count ─────────────────────────────────────────
    try:
        stats = cognito.describe_user_pool(UserPoolId=pool_id)["UserPool"]
        estimated_users = stats.get("EstimatedNumberOfUsers", "Unknown")
    except Exception:
        estimated_users = "Unknown"
    results.append(AuditResult(
        check="Estimated Users",
        status="INFO",
        detail=f"~{estimated_users} users — larger pools may take longer to initially sync",
        action_required="Allow additional time for initial replication if user count is large"
    ))

    return results


def print_audit_report(results: List[AuditResult], pool_id: str):
    icons = {"PASS": "✅", "FAIL": "❌", "WARN": "⚠️ ", "INFO": "ℹ️ "}
    print(f"\n{'='*65}")
    print(f"  COGNITO MRR MIGRATION READINESS AUDIT: {pool_id}")
    print(f"{'='*65}")

    blockers = [r for r in results if r.status == "FAIL"]
    warnings = [r for r in results if r.status == "WARN"]

    for r in results:
        print(f"\n{icons[r.status]} [{r.status}] {r.check}")
        print(f"   Detail : {r.detail}")
        if r.action_required:
            print(f"   Action : {r.action_required}")

    print(f"\n{'='*65}")
    print(f"  BLOCKERS: {len(blockers)}  |  WARNINGS: {len(warnings)}")
    print(f"  {'🚫 MIGRATION BLOCKED — Fix all FAIL items first.' if blockers else '🟢 Ready to proceed (review warnings).'}")
    print(f"{'='*65}\n")


if __name__ == "__main__":
    POOL_ID = sys.argv[1] if len(sys.argv) > 1 else "us-west-2_XXXXXXXXX"
    REGION  = sys.argv[2] if len(sys.argv) > 2 else "us-west-2"
    results = audit_pool_for_mrr(POOL_ID, REGION)
    print_audit_report(results, POOL_ID)

Usage:

python3 cognito_mrr_audit.py us-west-2_XXXXXXXXX us-west-2

Phase 2: Upgrade the Feature Plan (If Needed)

# Upgrade from Lite to Essentials
aws cognito-idp update-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --user-pool-tier ESSENTIALS

# Verify the upgrade
aws cognito-idp describe-user-pool \
  --region us-west-2 \
  --user-pool-id us-west-2_XXXXXXXXX \
  --query "UserPool.UserPoolTier"

Phase 3: Attach a Multi-Region KMS Key to an Existing Pool

⚠️ If your pool is already using an AWS managed key or a single-region CMK, you must create a new multi-region key. You cannot convert an existing single-region key to multi-region.

CLI: Create, Replicate, and Attach

PRIMARY_REGION="us-west-2"
SECONDARY_REGION="us-east-1"
POOL_ID="us-west-2_XXXXXXXXX"
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Step 1: Create multi-region primary key
MRK_KEY_ID=$(aws kms create-key \
  --region $PRIMARY_REGION \
  --multi-region \
  --description "Cognito MRR CMK" \
  --query "KeyMetadata.KeyId" \
  --output text)

echo "Primary MRK Key ID: $MRK_KEY_ID"

MRK_KEY_ARN="arn:aws:kms:${PRIMARY_REGION}:${ACCOUNT_ID}:key/${MRK_KEY_ID}"

# Step 2: Apply key policy (must include identitystore.amazonaws.com for replication)
cat > /tmp/key-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowRoot",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::${ACCOUNT_ID}:root"},
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "AllowCognitoAndIdentityStore",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "cognito-idp.amazonaws.com",
          "identitystore.amazonaws.com"
        ]
      },
      "Action": [
        "kms:Encrypt","kms:Decrypt","kms:ReEncrypt*",
        "kms:GenerateDataKey*","kms:DescribeKey","kms:CreateGrant"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {"aws:SourceAccount": "${ACCOUNT_ID}"}
      }
    }
  ]
}
EOF

# Apply policy to primary key
aws kms put-key-policy \
  --region $PRIMARY_REGION \
  --key-id $MRK_KEY_ID \
  --policy-name default \
  --policy file:///tmp/key-policy.json

# Step 3: Replicate key to secondary region
aws kms replicate-key \
  --region $PRIMARY_REGION \
  --key-id $MRK_KEY_ARN \
  --replica-region $SECONDARY_REGION

# Wait for the replica key to become active
echo "Waiting for replica key to become active..."
sleep 15

# Step 4: Apply the same policy to the replica key
aws kms put-key-policy \
  --region $SECONDARY_REGION \
  --key-id $MRK_KEY_ARN \   # Multi-region keys share same ARN prefix
  --policy-name default \
  --policy file:///tmp/key-policy.json

# Step 5: Attach the multi-region KMS key to the existing user pool
aws cognito-idp update-user-pool \
  --region $PRIMARY_REGION \
  --user-pool-id $POOL_ID \
  --kms-key-id $MRK_KEY_ARN

Phase 4: The Critical — Switch to Updated OIDC Issuer

This is the highest-risk step. The OIDC issuer change modifies the iss claim in all new tokens . Every backend service, API Gateway authorizer, and JWT validator that checks iss will break if not updated first.

Before Switching — Find All Affected Services

# Search CloudFormation stacks for Cognito issuer references
aws cloudformation list-stacks \
  --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE \
  --query "StackSummaries[].StackName" \
  --output text | tr '\t' '\n' | while read stack; do
    aws cloudformation get-template --stack-name "$stack" 2>/dev/null | \
      grep -l "cognito-idp\." && echo "  ↳ Found in: $stack"
  done

# Check API Gateway authorizers
aws apigateway get-rest-apis --query "items[].id" --output text | \
  tr '\t' '\n' | while read api_id; do
    aws apigateway get-authorizers --rest-api-id $api_id \
      --query "items[?type=='COGNITO_USER_POOLS'].{name:name,uri:authorizerUri}" \
      --output table 2>/dev/null
  done

Issuer URL Change Reference

	Old Issuer (Legacy)	New Issuer (Updated)
Format	`https://cognito-idp.{region}.amazonaws.com/{poolId}`	`https://issuer-cognito-idp.{region}.amazonaws.com/{poolId}`
JWKS Endpoint	`.../..well-known/jwks.json`	Same path, new base URL
Breaking?	Current	✅ Yes — update apps before switching
Token `iss` claim	Per-region	Same for both primary and replica

CLI: Perform the Issuer Switch

# ⚠️  Only run this AFTER updating all downstream JWT validators

# ⚠️  IMPORTANT: Switching to the multi-region OIDC issuer ("Updated" issuer type)
# is a CONSOLE-ONLY operation — it cannot be performed via the CLI or SDK.
# Navigate to: Cognito Console → User Pool → App Integration → Issuer URL → Change issuer type

# After switching via the console, verify the change:
aws cognito-idp describe-user-pool \
  --region $PRIMARY_REGION \
  --user-pool-id $POOL_ID \
  --query "UserPool.IssuerConfiguration"

Python: Bulk-Update API Gateway Authorizers

#!/usr/bin/env python3
"""
update_apigw_authorizers.py
Finds all API Gateway Cognito authorizers and updates the
issuer URL from the old legacy format to the new Updated format.
"""

import boto3
import re

REGION        = "us-west-2"
OLD_ISS_PREFIX = "https://cognito-idp."
NEW_ISS_PREFIX = "https://issuer-cognito-idp."

apigw  = boto3.client("apigateway", region_name=REGION)
apigwv2 = boto3.client("apigatewayv2", region_name=REGION)

def update_rest_api_authorizers():
    """Update REST API Cognito authorizers."""
    apis = apigw.get_rest_apis()["items"]
    for api in apis:
        api_id   = api["id"]
        api_name = api["name"]
        authorizers = apigw.get_authorizers(restApiId=api_id).get("items", [])

        for auth in authorizers:
            if auth.get("type") != "COGNITO_USER_POOLS":
                continue

            provider_arns = auth.get("providerARNs", [])
            print(f"\n🔍 API: {api_name} ({api_id}) | Authorizer: {auth['name']}")

            # The issuer is embedded in the userPoolArn — patch the audience/config
            # For REST APIs, update the JWT issuer in policy or Lambda authorizer config
            # For Cognito-native authorizers, the issuer is inferred from the user pool ARN
            # No direct issuer string to patch here — update your custom JWT validators instead
            print(f"   Provider ARNs: {provider_arns}")
            print(f"   ℹ️  Cognito-native authorizers use pool ARN — no direct issuer string to patch.")
            print(f"   ✅ These will automatically use the new issuer once pool is updated.")


def update_http_api_authorizers():
    """Update HTTP API JWT authorizers (explicitly reference issuer URL)."""
    apis = apigwv2.get_apis()["Items"]
    for api in apis:
        api_id   = api["ApiId"]
        api_name = api["Name"]
        authorizers = apigwv2.get_authorizers(ApiId=api_id).get("Items", [])

        for auth in authorizers:
            if auth.get("AuthorizerType") != "JWT":
                continue

            jwt_config   = auth.get("JwtConfiguration", {})
            current_issuer = jwt_config.get("Issuer", "")

            if OLD_ISS_PREFIX in current_issuer:
                new_issuer = current_issuer.replace(OLD_ISS_PREFIX, NEW_ISS_PREFIX)
                print(f"\n🔧 Updating HTTP API: {api_name} ({api_id})")
                print(f"   Auth: {auth['Name']}")
                print(f"   OLD issuer: {current_issuer}")
                print(f"   NEW issuer: {new_issuer}")

                apigwv2.update_authorizer(
                    ApiId=api_id,
                    AuthorizerId=auth["AuthorizerId"],
                    JwtConfiguration={
                        "Issuer":   new_issuer,
                        "Audience": jwt_config.get("Audience", [])
                    }
                )
                print(f"   ✅ Updated.")
            else:
                print(f"\n✅ HTTP API: {api_name} ({api_id}) | {auth['Name']} — already updated or not Cognito")


if __name__ == "__main__":
    print("=" * 60)
    print("  Scanning REST API Authorizers")
    print("=" * 60)
    update_rest_api_authorizers()

    print("\n" + "=" * 60)
    print("  Scanning HTTP API JWT Authorizers")
    print("=" * 60)
    update_http_api_authorizers()

    print("\n✅ Scan complete. Review any remaining custom JWT validators in your application code.")

Phase 5: Create the Replica and Activate

Once the pool has a multi-region KMS key and an updated OIDC issuer, creating the replica is straightforward .

# Create the replica (call is made against the PRIMARY region)
aws cognito-idp create-user-pool-replica-region \
  --region $PRIMARY_REGION \
  --user-pool-id $POOL_ID \
  --replica-region '{"RegionName": "'$SECONDARY_REGION'", "KmsKeyId": "'$MRK_KEY_ARN'"}'

# Poll status — replica info lives on the primary pool's ReplicaRegions field
watch -n 15 "aws cognito-idp describe-user-pool \
  --user-pool-id $POOL_ID \
  --region $PRIMARY_REGION \
  --query 'UserPool.ReplicaRegions[*].{Region:RegionName,Status:Status}' \
  --output table"

# Once ACTIVE, get the replica pool ID from the secondary region directly
REPLICA_POOL_ID=$(aws cognito-idp describe-user-pool \
  --region $SECONDARY_REGION \
  --user-pool-id "${POOL_ID/$PRIMARY_REGION/$SECONDARY_REGION}" \
  --query "UserPool.Id" \
  --output text)

# Configure replica-specific Lambda triggers (must point to secondary-region functions)
aws cognito-idp update-user-pool \
  --region $SECONDARY_REGION \
  --user-pool-id $REPLICA_POOL_ID \
  --lambda-config '{
    "PostAuthentication": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:cognito-post-auth",
    "PreTokenGeneration": "arn:aws:lambda:us-east-1:<ACCOUNT_ID>:function:cognito-pre-token"
  }'

echo "✅ Replica is ACTIVE in $SECONDARY_REGION"

Phase 6: Configure Failover-Aware Application Code

After MRR is enabled, your app must intelligently route write vs. read operations and handle failover.

Python: Smart Cognito Client with Regional Failover

#!/usr/bin/env python3
"""
cognito_smart_client.py
A resilient Cognito client that:
- Routes writes to primary region
- Routes reads (sign-in) to nearest healthy region
- Falls back automatically on OperationNotEnabledException
"""

import boto3
import logging
from botocore.exceptions import ClientError
from typing import Optional

log = logging.getLogger(__name__)

PRIMARY_REGION   = "us-west-2"
SECONDARY_REGION = "us-east-1"
CLIENT_ID        = "your-app-client-id"


class ResilientCognitoClient:
    def __init__(self, prefer_secondary: bool = False):
        self._primary   = boto3.client("cognito-idp", region_name=PRIMARY_REGION)
        self._secondary = boto3.client("cognito-idp", region_name=SECONDARY_REGION)

        # Determine which region to use for authentication
        self._auth_client = self._secondary if prefer_secondary else self._primary
        self._auth_region = SECONDARY_REGION if prefer_secondary else PRIMARY_REGION

    # ── READ OPERATIONS (replica-safe) ──────────────────────
    def sign_in(self, username: str, password: str) -> dict:
        """
        Attempts sign-in on preferred region; falls back to primary if replica is degraded.
        """
        for client, region in [
            (self._auth_client, self._auth_region),
            (self._primary, PRIMARY_REGION)
        ]:
            try:
                resp = client.initiate_auth(
                    AuthFlow="USER_PASSWORD_AUTH",
                    ClientId=CLIENT_ID,
                    AuthParameters={"USERNAME": username, "PASSWORD": password}
                )
                log.info(f"✅ Authenticated via {region}")
                return resp["AuthenticationResult"]
            except ClientError as e:
                code = e.response["Error"]["Code"]
                if code in ("OperationNotEnabledException", "ServiceUnavailableException"):
                    log.warning(f"⚠️  {region} unavailable ({code}), trying fallback...")
                    continue
                raise  # Re-raise auth errors (wrong password, etc.)

        raise RuntimeError("Authentication failed in all regions")

    def get_user(self, access_token: str) -> dict:
        """Token was issued by either region — try both if needed."""
        for client, region in [
            (self._auth_client, self._auth_region),
            (self._primary, PRIMARY_REGION)
        ]:
            try:
                return client.get_user(AccessToken=access_token)
            except ClientError as e:
                if "NotAuthorizedException" in e.response["Error"]["Code"]:
                    raise  # Bad token — don't retry
                log.warning(f"get_user failed on {region}: {e}")
                continue
        raise RuntimeError("get_user failed in all regions")

    # ── WRITE OPERATIONS (primary only) ──────────────────────
    def sign_up(self, username: str, password: str, email: str) -> dict:
        """Always routes to primary — writes are rejected on replica."""
        return self._primary.sign_up(
            ClientId=CLIENT_ID,
            Username=username,
            Password=password,
            UserAttributes=[{"Name": "email", "Value": email}]
        )

    def change_password(self, access_token: str, old_pw: str, new_pw: str):
        """Must go to primary — OperationNotEnabledException on replica."""
        return self._primary.change_password(
            AccessToken=access_token,
            PreviousPassword=old_pw,
            ProposedPassword=new_pw
        )

    def forgot_password(self, username: str):
        """Password reset always to primary."""
        return self._primary.forgot_password(
            ClientId=CLIENT_ID,
            Username=username
        )

    def refresh_tokens(self, refresh_token: str) -> dict:
        """Refresh works on both regions; try preferred first."""
        for client, region in [
            (self._auth_client, self._auth_region),
            (self._primary, PRIMARY_REGION)
        ]:
            try:
                resp = client.initiate_auth(
                    AuthFlow="REFRESH_TOKEN_AUTH",
                    ClientId=CLIENT_ID,
                    AuthParameters={"REFRESH_TOKEN": refresh_token}
                )
                log.info(f"✅ Token refreshed via {region}")
                return resp["AuthenticationResult"]
            except ClientError as e:
                if e.response["Error"]["Code"] == "NotAuthorizedException":
                    raise  # Invalid/expired refresh token
                log.warning(f"Refresh failed on {region}: {e}")
                continue

        raise RuntimeError("Token refresh failed in all regions")


# ── Usage Example ────────────────────────────────────────────
if __name__ == "__main__":
    client = ResilientCognitoClient(prefer_secondary=False)  # Set True during failover

    try:
        tokens = client.sign_in("testuser@example.com", "MyPassword123!")
        print("Access Token:", tokens["AccessToken"][:20], "...")
        print("ID Token:    ", tokens["IdToken"][:20], "...")
    except RuntimeError as e:
        print(f"❌ Auth failed: {e}")

Terraform: Migrating an Existing Pool (State Import + MRR Attachment)

If you manage your existing pool in Terraform but it was created before MRR, use terraform import to bring it under the new MRR-enabled config without re-creating users.

# terraform.tf — Update existing pool resource block to add MRR config

resource "aws_cognito_user_pool" "existing" {
  provider = aws.primary
  name     = "myapp-user-pool"       # Keep the same name

  # ── Add these new blocks to existing resource ─────────────
  user_pool_tier = "ESSENTIALS"       # Upgrade from LITE if needed

  # Attach multi-region KMS key
  # (aws_kms_key.cognito_mrk created as shown in previous blog post)

  tags = {
    Environment = "production"
    MRR         = "enabled"
  }
}

# After terraform apply, create the replica
resource "aws_cognito_user_pool_replica" "secondary" {
  provider     = aws.primary
  user_pool_id = aws_cognito_user_pool.existing.id
  region_name  = "us-east-1"

  depends_on = [
    aws_kms_replica_key.cognito_mrk_replica
  ]
}

# Import existing pool into Terraform state (no re-creation of users)
terraform import aws_cognito_user_pool.existing us-west-2_XXXXXXXXX

# Plan — verify only MRR-related changes, not destructive ones
terraform plan -out=mrr-migration.tfplan

# Review the plan carefully — ensure no "destroy" on user_pool
grep -i "destroy\|replace" mrr-migration.tfplan

# Apply when satisfied
terraform apply mrr-migration.tfplan

Full Migration Sequence Summary

Step	Action	Risk	CLI Command
0	Check next-gen eligibility	Low	`list-user-pool-replicas`
1	Upgrade to Essentials/Plus	Low	`update-user-pool --user-pool-tier ESSENTIALS`
2	Create multi-region KMS key	Low	`kms create-key --multi-region`
3	Replicate KMS key to secondary	Low	`kms replicate-key`
4	Attach KMS key to user pool	Low	`update-user-pool --key-configuration`
5	Update all JWT validators to new issuer	HIGH	Manual app update
6	Switch OIDC issuer to `UPDATED`	HIGH	`update-user-pool --issuer-configuration Type=UPDATED`
7	Configure replica-specific settings	Medium	`update-user-pool` on secondary
8	Create replica	Low	`create-user-pool-replica`
9	Wait for replica to become `ACTIVE`	Low	`describe-user-pool --query UserPool.ReplicaRegions`
10	Set up Route 53 health check	Low	`route53 create-health-check`
11	Enable failover on custom domain	Low	Console or CLI
12	Test failover in staging	Medium	`initiate-auth` against replica region

Post-Migration Verification

# Verify replica is syncing users correctly
REPLICA_POOL_ID="us-east-1_XXXXXXXXX"  # Get from list-user-pool-replicas

# Check a known user on the replica
aws cognito-idp admin-get-user \
  --region $SECONDARY_REGION \
  --user-pool-id $REPLICA_POOL_ID \
  --username testuser@example.com

# Test auth on the replica region
aws cognito-idp initiate-auth \
  --region $SECONDARY_REGION \
  --auth-flow USER_PASSWORD_AUTH \
  --client-id $CLIENT_ID \
  --auth-parameters USERNAME=testuser@example.com,PASSWORD='TestPass123!'

# Confirm the iss claim matches the updated issuer format
REPLICA_TOKEN=$(aws cognito-idp initiate-auth \
  --region $SECONDARY_REGION \
  --auth-flow USER_PASSWORD_AUTH \
  --client-id $CLIENT_ID \
  --auth-parameters USERNAME=testuser@example.com,PASSWORD='TestPass123!' \
  --query "AuthenticationResult.IdToken" \
  --output text)

# Decode and check the iss claim (requires jq and base64)
echo $REPLICA_TOKEN | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '{iss: .iss, sub: .sub, aud: .aud}'

The iss claim should now read https://issuer-cognito-idp.us-west-2.amazonaws.com/us-west-2_XXXXXXXXX — identical for tokens from both regions, confirming that your JWT validators need no per-region branching .

AWS Global View: The Ultimate Multi-Region Resource Visibility Guide (CLI + Terraform + CDK + Python)

Manish Kumar — Thu, 28 May 2026 07:21:40 +0000

Introduction

If you manage AWS infrastructure across multiple regions — which every production-grade setup does — you've felt the pain of jumping between region tabs in the console just to check how many EC2 instances are running globally, or hunting down an orphaned EIP you forgot about in ap-southeast-1. AWS Global View solves exactly that problem.

AWS Global View is a read-only, single-pane-of-glass console built directly into the EC2/VPC console at us-east-1 that gives you a unified summary of EC2 and VPC resources across every AWS region your account is enabled in. It requires zero setup, zero cost, and zero configuration. Yet, most AWS engineers don't know it exists — and even fewer know how to replicate and extend its capabilities programmatically using CLI, Terraform, CDK, and boto3.

In this blog, we cover every angle: from the console deep-dive to full IaC automation and Python-based global inventory scripts you can drop straight into production.

What is AWS Global View?

AWS Global View is a read-only multi-region resource explorer embedded inside the AWS EC2 console. It was designed to give you a bird's-eye view of your EC2 and VPC fleet without switching regions. It does NOT allow you to create, modify, or delete resources — it is purely an observation tool.

The console is accessible at:

https://<account-id>.us-east-1.console.aws.amazon.com/awsglobalview/home?region=us-east-1#RegionExplorer

Pro Tip: AWS Global View only works in us-east-1 as the home region. You cannot access it from other regional console endpoints. Also, Firefox private/incognito windows are not supported.

The service is tightly integrated with AWS Resource Explorer v2, which provides the underlying index and search engine powering global search features .

Supported Resources

AWS Global View currently supports the following resources across all enabled regions:

Category	Resources
Compute	EC2 Instances, Auto Scaling Groups, Capacity Reservations, Capacity Blocks
Networking	VPCs, Subnets, Internet Gateways, Egress-only IGWs, NAT Gateways, Route Tables, Network ACLs, Network Interfaces, Elastic IPs, Security Groups
Connectivity	VPC Endpoints, VPC Peering Connections, Endpoint Services, Managed Prefix Lists
Storage	EBS Volumes, S3 Buckets
Database	RDS DB Instances, DB Clusters
Infrastructure	DHCP Option Sets, Outposts, Availability Zones

Console Deep Dive

The Global View console has four key sections:

Region Explorer

The landing page. It shows a Summary section at the top — total count of each resource type across all enabled regions, with clickable links. For example, if it says "29 Instances in 10 Regions", clicking that link shows all 29 with their regions listed. Below the summary is a per-region breakdown table — every region as a row, every resource type as a column, with counts. Click any cell to drill into that specific resource type in that region.

Global Search

The search tab lets you filter by Region, Resource Type, and Tags simultaneously. This is the closest thing AWS has to a native cross-region asset inventory. You can click any resource ID to jump directly into its native console (e.g., clicking an instance ID opens it in the EC2 console for that specific region).

Regions and Zones

This tab lists all AWS Regions, Availability Zones, Local Zones, and Wavelength Zones for your account. From here you can enable/disable regions and opt-in to Local Zones — critical for compliance use cases where you need to restrict infrastructure to specific geographies.

Settings

Customize the console to hide resource types and regions you don't use. If your organization only uses us-east-1, ap-south-1, and eu-west-1, hide everything else to reduce noise.

Required IAM Permissions

A user needs read-level ec2:Describe* and rds:Describe* permissions at minimum to access Global View. Here is the least-privilege policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlobalViewReadOnly",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeRegions",
        "ec2:DescribeInstances",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:DescribeNatGateways",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeRouteTables",
        "ec2:DescribeNetworkAcls",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeAddresses",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeVpcPeeringConnections",
        "ec2:DescribeVpcEndpoints",
        "ec2:DescribeManagedPrefixLists",
        "ec2:DescribeAutoScalingGroups",
        "rds:DescribeDBInstances",
        "rds:DescribeDBClusters",
        "s3:ListAllMyBuckets",
        "account:GetRegionOptStatus",
        "account:ListRegions"
      ],
      "Resource": "*"
    }
  ]
}

Security Best Practice: Attach this policy to an IAM Role, not an IAM User. If you're using this from a Lambda function or GitHub Actions CI/CD runner, use role assumption with STS — never hardcode access keys.

Section 1: AWS CLI — Replicate Global View in the Terminal

AWS Global View has no dedicated CLI command. Instead, you replicate its behavior using ec2 and account CLI subcommands. All examples below assume you have the AWS CLI v2 configured with appropriate credentials.

1.1 — List All Enabled Regions

aws ec2 describe-regions \
  --filters "Name=opt-in-status,Values=opt-in-not-required,opted-in" \
  --query "Regions[*].RegionName" \
  --output table

This gives you the exact list Global View uses to build its region table.

1.2 — Count EC2 Instances Per Region

echo "Region | Instance Count" && echo "-------|---------------"
for region in $(aws ec2 describe-regions \
  --filters "Name=opt-in-status,Values=opt-in-not-required,opted-in" \
  --query "Regions[*].RegionName" --output text); do
  count=$(aws ec2 describe-instances \
    --region "$region" \
    --query "length(Reservations[*].Instances[])" \
    --output text 2>/dev/null || echo 0)
  echo "$region | $count"
done

1.3 — List All VPCs Globally with CIDR Blocks

for region in $(aws ec2 describe-regions \
  --query "Regions[*].RegionName" --output text); do
  echo "=== Region: $region ==="
  aws ec2 describe-vpcs \
    --region "$region" \
    --query "Vpcs[*].{VpcId:VpcId,CIDR:CidrBlock,Default:IsDefault,State:State}" \
    --output table 2>/dev/null
done

1.4 — Find All Unattached EBS Volumes (FinOps Use Case)

Orphaned EBS volumes are one of the biggest sources of wasted AWS spend. This script finds them globally:

echo "Hunting orphaned volumes across all regions..."
for region in $(aws ec2 describe-regions \
  --query "Regions[*].RegionName" --output text); do
  result=$(aws ec2 describe-volumes \
    --region "$region" \
    --filters "Name=status,Values=available" \
    --query "Volumes[*].{Region:'$region',VolumeId:VolumeId,Size:Size,Type:VolumeType}" \
    --output table 2>/dev/null)
  if [[ -n "$result" && "$result" != *"None"* ]]; then
    echo "=== $region ==="
    echo "$result"
  fi
done

1.5 — Find Open Security Groups (0.0.0.0/0 on Port 22 or 3389)

This is a global security audit you'd normally do manually per-region:

for region in $(aws ec2 describe-regions \
  --query "Regions[*].RegionName" --output text); do
  aws ec2 describe-security-groups \
    --region "$region" \
    --filters "Name=ip-permission.from-port,Values=22" \
             "Name=ip-permission.cidr,Values=0.0.0.0/0" \
    --query "SecurityGroups[*].{Region:'$region',GroupId:GroupId,GroupName:GroupName,VpcId:VpcId}" \
    --output table 2>/dev/null
done

1.6 — Enable/Disable a Region via CLI

# Check region opt-in status
aws account get-region-opt-status \
  --region-name ap-east-1 \
  --region us-east-1

# Enable a region (opt-in regions like ap-east-1, me-south-1)
aws account enable-region \
  --region-name ap-east-1 \
  --region us-east-1

# Disable a region
aws account disable-region \
  --region-name ap-east-1 \
  --region us-east-1

Important: Region enable/disable operations require the account:EnableRegion and account:DisableRegion IAM permissions, and must be called against the us-east-1 endpoint regardless of target region.

Section 2: Resource Explorer v2 CLI — The API Behind Global Search

AWS Resource Explorer v2 is the underlying engine that powers the Global Search tab in Global View . Unlike Global View (which is console-only), Resource Explorer has a full API and CLI.

2.1 — Enable Resource Explorer with an Aggregator Index

The aggregator index is the "master" that collects data from local indexes in all other regions :

# Step 1: Create local indexes in each region you want to cover
for region in ap-south-1 eu-west-1 us-west-2 ap-southeast-1; do
  echo "Creating local index in $region..."
  aws resource-explorer-2 create-index \
    --type LOCAL \
    --region "$region"
done

# Step 2: Create the aggregator index in us-east-1
aws resource-explorer-2 create-index \
  --type AGGREGATOR \
  --region us-east-1

If you already have a local index in us-east-1, upgrade it to AGGREGATOR:

# Get the index ARN first
aws resource-explorer-2 get-index --region us-east-1

# Then upgrade it
aws resource-explorer-2 update-index-type \
  --arn "arn:aws:resource-explorer-2:us-east-1:123456789012:index/YOUR-INDEX-ID" \
  --type AGGREGATOR \
  --region us-east-1

2.2 — Create Search Views

Views are saved search filters — like a named query for cross-region searches :

# Create a view for all resources (shows all tags)
aws resource-explorer-2 create-view \
  --view-name "All-Resources-Global" \
  --included-properties Name=tags \
  --region us-east-1

# Create an EC2-only view
aws resource-explorer-2 create-view \
  --view-name "EC2-Only-Global" \
  --included-properties Name=tags \
  --filters FilterString="service:ec2" \
  --region us-east-1

# Set a view as default for the region
aws resource-explorer-2 associate-default-view \
  --view-arn "arn:aws:resource-explorer-2:us-east-1:123456789012:view/All-Resources-Global/YOUR-VIEW-ID"

2.3 — Search Resources Globally via CLI

# Search all EC2 instances globally
aws resource-explorer-2 search \
  --query-string "resourcetype:ec2:instance" \
  --region us-east-1

# Search by tag (e.g., Environment=Production)
aws resource-explorer-2 search \
  --query-string "tag.Environment=Production" \
  --region us-east-1

# Search for all untagged resources
aws resource-explorer-2 search \
  --query-string "-tag.Environment" \
  --region us-east-1

# Search for RDS instances across all regions
aws resource-explorer-2 search \
  --query-string "service:rds resourcetype:rds:db" \
  --region us-east-1

# Use a specific view for EC2-scoped search
aws resource-explorer-2 search \
  --query-string "*" \
  --view-arn "arn:aws:resource-explorer-2:us-east-1:123456789012:view/EC2-Only-Global/YOUR-VIEW-ID" \
  --region us-east-1

2.4 — List All Resource Types Supported by Resource Explorer

# Get all supported resource types (paginated)
aws resource-explorer-2 list-supported-resource-types \
  --max-items 50 \
  --region us-east-1

# Get next page using the NextToken
aws resource-explorer-2 list-supported-resource-types \
  --max-items 50 \
  --starting-token "NEXT_TOKEN_HERE" \
  --region us-east-1

Section 3: Infrastructure as Code — Terraform

3.1 — Multi-Region Provider Setup

Always define providers as aliases in Terraform when working across multiple regions. Here's a production-grade providers.tf:

# providers.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }
}

# Default provider — us-east-1 (where Global View lives)
provider "aws" {
  region = "us-east-1"
}

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

provider "aws" {
  alias  = "us_west_2"
  region = "us-west-2"
}

provider "aws" {
  alias  = "ap_south_1"
  region = "ap-south-1"
}

provider "aws" {
  alias  = "eu_west_1"
  region = "eu-west-1"
}

3.2 — IAM Policy and Role for Global View Access

# iam.tf
resource "aws_iam_policy" "global_view_readonly" {
  name        = "GlobalViewReadOnlyPolicy"
  description = "Least-privilege read-only policy for AWS Global View and Resource Explorer"
  path        = "/"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "EC2GlobalViewAccess"
        Effect = "Allow"
        Action = [
          "ec2:DescribeRegions",
          "ec2:DescribeInstances",
          "ec2:DescribeVpcs",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeVolumes",
          "ec2:DescribeNatGateways",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeRouteTables",
          "ec2:DescribeNetworkAcls",
          "ec2:DescribeNetworkInterfaces",
          "ec2:DescribeAddresses",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeVpcPeeringConnections",
          "ec2:DescribeVpcEndpoints",
          "ec2:DescribeManagedPrefixLists",
          "ec2:DescribeAutoScalingGroups"
        ]
        Resource = "*"
      },
      {
        Sid    = "RDSGlobalViewAccess"
        Effect = "Allow"
        Action = [
          "rds:DescribeDBInstances",
          "rds:DescribeDBClusters"
        ]
        Resource = "*"
      },
      {
        Sid    = "S3GlobalViewAccess"
        Effect = "Allow"
        Action = ["s3:ListAllMyBuckets", "s3:GetBucketLocation"]
        Resource = "*"
      },
      {
        Sid    = "ResourceExplorerAccess"
        Effect = "Allow"
        Action = [
          "resource-explorer-2:Search",
          "resource-explorer-2:GetIndex",
          "resource-explorer-2:ListIndexes",
          "resource-explorer-2:ListViews",
          "resource-explorer-2:GetView",
          "resource-explorer-2:ListSupportedResourceTypes"
        ]
        Resource = "*"
      },
      {
        Sid    = "AccountRegionAccess"
        Effect = "Allow"
        Action = [
          "account:GetRegionOptStatus",
          "account:ListRegions"
        ]
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role" "global_view_role" {
  name        = "GlobalViewReadOnlyRole"
  description = "IAM Role for Global View access - Lambda or CI/CD"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = [
            "lambda.amazonaws.com",
            "ec2.amazonaws.com"
          ]
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "global_view_attach" {
  role       = aws_iam_role.global_view_role.name
  policy_arn = aws_iam_policy.global_view_readonly.arn
}

output "global_view_role_arn" {
  value       = aws_iam_role.global_view_role.arn
  description = "ARN of the Global View read-only IAM role"
}

3.3 — Terraform: Enable Resource Explorer in All Regions

# resource_explorer.tf

# Create the AGGREGATOR index in us-east-1
resource "aws_resourceexplorer2_index" "aggregator" {
  provider = aws.us_east_1
  type     = "AGGREGATOR"

  tags = {
    Name        = "global-aggregator-index"
    ManagedBy   = "Terraform"
    Environment = "production"
  }
}

# Local indexes in each region (must exist before aggregator can pull)
resource "aws_resourceexplorer2_index" "ap_south_1" {
  provider = aws.ap_south_1
  type     = "LOCAL"

  tags = {
    Name      = "local-index-ap-south-1"
    ManagedBy = "Terraform"
  }

  depends_on = [aws_resourceexplorer2_index.aggregator]
}

resource "aws_resourceexplorer2_index" "us_west_2" {
  provider = aws.us_west_2
  type     = "LOCAL"

  tags = {
    Name      = "local-index-us-west-2"
    ManagedBy = "Terraform"
  }

  depends_on = [aws_resourceexplorer2_index.aggregator]
}

3.4 — Terraform: Create Resource Explorer Views

# views.tf

resource "aws_resourceexplorer2_view" "all_resources" {
  provider = aws.us_east_1
  name     = "All-Resources-Global"

  included_property {
    name = "tags"
  }

  depends_on = [aws_resourceexplorer2_index.aggregator]

  tags = {
    ManagedBy = "Terraform"
  }
}

resource "aws_resourceexplorer2_view" "ec2_only" {
  provider = aws.us_east_1
  name     = "EC2-Only-Global"

  filters {
    filter_string = "service:ec2"
  }

  included_property {
    name = "tags"
  }

  depends_on = [aws_resourceexplorer2_index.aggregator]
}

# Set as default view in us-east-1
resource "aws_resourceexplorer2_default_view" "default" {
  provider = aws.us_east_1
  view_arn = aws_resourceexplorer2_view.all_resources.arn
}

output "global_view_arn" {
  value = aws_resourceexplorer2_view.all_resources.arn
}

3.5 — Terraform: Cross-Region Data Sources for Inventory

# inventory.tf

data "aws_instances" "us_east_1" {
  provider = aws.us_east_1
  filter {
    name   = "instance-state-name"
    values = ["running", "stopped"]
  }
}

data "aws_instances" "ap_south_1" {
  provider = aws.ap_south_1
  filter {
    name   = "instance-state-name"
    values = ["running", "stopped"]
  }
}

data "aws_vpcs" "us_east_1" {
  provider = aws.us_east_1
}

data "aws_vpcs" "ap_south_1" {
  provider = aws.ap_south_1
}

output "us_east_1_instances" {
  value = {
    ids   = data.aws_instances.us_east_1.ids
    count = length(data.aws_instances.us_east_1.ids)
  }
}

output "ap_south_1_instances" {
  value = {
    ids   = data.aws_instances.ap_south_1.ids
    count = length(data.aws_instances.ap_south_1.ids)
  }
}

output "global_vpc_summary" {
  value = {
    us_east_1_vpcs   = tolist(data.aws_vpcs.us_east_1.ids)
    ap_south_1_vpcs  = tolist(data.aws_vpcs.ap_south_1.ids)
  }
}

Section 4: AWS CDK (Python) — Global View Infrastructure as Code

4.1 — Project Bootstrap

mkdir aws-global-view-cdk && cd aws-global-view-cdk
python3 -m venv .venv
source .venv/bin/activate

pip install aws-cdk-lib constructs boto3

cdk init app --language python

# Install the Resource Explorer L1 constructs
pip install aws-cdk.aws-resourceexplorer2

4.2 — CDK App Entry Point

# app.py
import aws_cdk as cdk
from stacks.global_view_stack import GlobalViewStack
from stacks.resource_explorer_stack import ResourceExplorerStack

app = cdk.App()

# Deploy IAM and Resource Explorer in us-east-1
GlobalViewStack(
    app,
    "GlobalViewStack",
    env=cdk.Environment(region="us-east-1"),
)

ResourceExplorerStack(
    app,
    "ResourceExplorerStack",
    env=cdk.Environment(region="us-east-1"),
)

app.synth()

4.3 — Global View IAM Stack

# stacks/global_view_stack.py
from aws_cdk import (
    Stack,
    aws_iam as iam,
    CfnOutput,
    Tags,
)
from constructs import Construct

class GlobalViewStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs):
        super().__init__(scope, construct_id, **kwargs)

        # Read-only policy for Global View
        policy = iam.ManagedPolicy(
            self,
            "GlobalViewPolicy",
            managed_policy_name="GlobalViewReadOnlyPolicy",
            statements=[
                iam.PolicyStatement(
                    sid="EC2GlobalView",
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "ec2:DescribeRegions",
                        "ec2:DescribeInstances",
                        "ec2:DescribeVpcs",
                        "ec2:DescribeSubnets",
                        "ec2:DescribeSecurityGroups",
                        "ec2:DescribeVolumes",
                        "ec2:DescribeNatGateways",
                        "ec2:DescribeInternetGateways",
                        "ec2:DescribeRouteTables",
                        "ec2:DescribeNetworkAcls",
                        "ec2:DescribeAddresses",
                        "ec2:DescribeAvailabilityZones",
                        "ec2:DescribeVpcPeeringConnections",
                        "ec2:DescribeVpcEndpoints",
                    ],
                    resources=["*"],
                ),
                iam.PolicyStatement(
                    sid="RDSAndS3Access",
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "rds:DescribeDBInstances",
                        "rds:DescribeDBClusters",
                        "s3:ListAllMyBuckets",
                        "s3:GetBucketLocation",
                    ],
                    resources=["*"],
                ),
                iam.PolicyStatement(
                    sid="ResourceExplorerSearch",
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "resource-explorer-2:Search",
                        "resource-explorer-2:GetIndex",
                        "resource-explorer-2:ListIndexes",
                        "resource-explorer-2:ListViews",
                        "resource-explorer-2:GetView",
                    ],
                    resources=["*"],
                ),
                iam.PolicyStatement(
                    sid="AccountAccess",
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "account:GetRegionOptStatus",
                        "account:ListRegions",
                    ],
                    resources=["*"],
                ),
            ],
        )

        # IAM Role for Lambda or automation to use Global View
        role = iam.Role(
            self,
            "GlobalViewRole",
            role_name="GlobalViewReadOnlyRole",
            assumed_by=iam.CompositePrincipal(
                iam.ServicePrincipal("lambda.amazonaws.com"),
                iam.ServicePrincipal("ec2.amazonaws.com"),
            ),
            managed_policies=[
                policy,
                iam.ManagedPolicy.from_aws_managed_policy_name(
                    "service-role/AWSLambdaBasicExecutionRole"
                ),
            ],
        )

        Tags.of(self).add("ManagedBy", "CDK")
        Tags.of(self).add("Purpose", "GlobalView")

        CfnOutput(self, "RoleArn", value=role.role_arn)
        CfnOutput(self, "PolicyArn", value=policy.managed_policy_arn)

4.4 — Resource Explorer CDK Stack

# stacks/resource_explorer_stack.py
from aws_cdk import (
    Stack,
    aws_resourceexplorer2 as rex,
    CfnOutput,
)
from constructs import Construct

class ResourceExplorerStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs):
        super().__init__(scope, construct_id, **kwargs)

        # Aggregator index in us-east-1
        aggregator_index = rex.CfnIndex(
            self,
            "AggregatorIndex",
            type="AGGREGATOR",
            tags=[{"key": "ManagedBy", "value": "CDK"}],
        )

        # Global view — all resources with tags
        global_view = rex.CfnView(
            self,
            "AllResourcesView",
            view_name="All-Resources-Global-CDK",
            included_properties=[
                rex.CfnView.IncludedPropertyProperty(name="tags")
            ],
            depends_on=[aggregator_index],
        )

        # EC2-scoped view
        ec2_view = rex.CfnView(
            self,
            "EC2OnlyView",
            view_name="EC2-Only-Global-CDK",
            filters=rex.CfnView.FiltersProperty(
                filter_string="service:ec2"
            ),
            included_properties=[
                rex.CfnView.IncludedPropertyProperty(name="tags")
            ],
            depends_on=[aggregator_index],
        )

        # Set default view
        rex.CfnDefaultViewAssociation(
            self,
            "DefaultView",
            view_arn=global_view.attr_view_arn,
        )

        CfnOutput(self, "AggregatorIndexArn",
                  value=aggregator_index.attr_arn)
        CfnOutput(self, "GlobalViewArn",
                  value=global_view.attr_view_arn)
        CfnOutput(self, "EC2ViewArn",
                  value=ec2_view.attr_view_arn)

Section 5: Python (boto3) — Build Your Own Global View

5.1 — Get All Enabled Regions

import boto3

def get_enabled_regions() -> list[str]:
    """Returns all enabled (opted-in or default) regions."""
    ec2 = boto3.client("ec2", region_name="us-east-1")
    response = ec2.describe_regions(
        Filters=[{
            "Name": "opt-in-status",
            "Values": ["opt-in-not-required", "opted-in"]
        }]
    )
    return sorted([r["RegionName"] for r in response["Regions"]])

if __name__ == "__main__":
    regions = get_enabled_regions()
    print(f"Enabled regions ({len(regions)}): {regions}")

5.2 — Multi-Threaded Global EC2 Inventory

Using ThreadPoolExecutor mimics how Global View fetches all regions in parallel — critical for performance with 20+ regions:

import boto3
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime

def get_instances_in_region(region: str) -> list[dict]:
    """Paginated EC2 instance fetch for a single region."""
    ec2 = boto3.client("ec2", region_name=region)
    paginator = ec2.get_paginator("describe_instances")
    instances = []
    try:
        for page in paginator.paginate():
            for reservation in page["Reservations"]:
                for inst in reservation["Instances"]:
                    # Extract Name tag
                    name = next(
                        (t["Value"] for t in inst.get("Tags", [])
                         if t["Key"] == "Name"),
                        "N/A"
                    )
                    instances.append({
                        "Region": region,
                        "InstanceId": inst["InstanceId"],
                        "Name": name,
                        "State": inst["State"]["Name"],
                        "InstanceType": inst["InstanceType"],
                        "PrivateIp": inst.get("PrivateIpAddress", "N/A"),
                        "PublicIp": inst.get("PublicIpAddress", "N/A"),
                        "LaunchTime": inst["LaunchTime"].strftime("%Y-%m-%d %H:%M:%S"),
                        "VpcId": inst.get("VpcId", "N/A"),
                    })
    except Exception as e:
        print(f"[ERROR] {region}: {e}")
    return instances

def global_ec2_inventory(max_workers: int = 15) -> list[dict]:
    regions = get_enabled_regions()
    all_instances = []
    print(f"Scanning {len(regions)} regions with {max_workers} threads...")

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        future_map = {
            executor.submit(get_instances_in_region, r): r
            for r in regions
        }
        for future in as_completed(future_map):
            region = future_map[future]
            result = future.result()
            count = len(result)
            if count > 0:
                print(f"  ✅ {region}: {count} instance(s)")
            else:
                print(f"  ⬜ {region}: 0 instances")
            all_instances.extend(result)

    return all_instances

if __name__ == "__main__":
    inventory = global_ec2_inventory()
    print(f"\nTotal instances found: {len(inventory)}")

5.3 — Full Global Resource Summary (Global View Equivalent)

This function replicates the "Summary" section at the top of the Region Explorer console:

import boto3
from concurrent.futures import ThreadPoolExecutor, as_completed

def get_region_summary(region: str) -> dict:
    """Collect resource counts for a single region."""
    ec2 = boto3.client("ec2", region_name=region)
    rds = boto3.client("rds", region_name=region)
    summary = {
        "Region": region,
        "Instances": 0,
        "VPCs": 0,
        "Volumes": 0,
        "SecurityGroups": 0,
        "Subnets": 0,
        "NatGateways": 0,
        "InternetGateways": 0,
        "ElasticIPs": 0,
        "DBInstances": 0,
        "Error": None,
    }
    try:
        summary["Instances"] = sum(
            len(r["Instances"])
            for r in ec2.describe_instances()["Reservations"]
        )
        summary["VPCs"] = len(ec2.describe_vpcs()["Vpcs"])
        summary["Volumes"] = len(ec2.describe_volumes()["Volumes"])
        summary["SecurityGroups"] = len(
            ec2.describe_security_groups()["SecurityGroups"]
        )
        summary["Subnets"] = len(ec2.describe_subnets()["Subnets"])
        summary["NatGateways"] = len(
            ec2.describe_nat_gateways(
                Filters=[{"Name": "state", "Values": ["available"]}]
            )["NatGateways"]
        )
        summary["InternetGateways"] = len(
            ec2.describe_internet_gateways()["InternetGateways"]
        )
        summary["ElasticIPs"] = len(
            ec2.describe_addresses()["Addresses"]
        )
        summary["DBInstances"] = len(
            rds.describe_db_instances()["DBInstances"]
        )
    except Exception as e:
        summary["Error"] = str(e)
    return summary

def generate_global_view_report() -> list[dict]:
    regions = get_enabled_regions()
    report = []
    with ThreadPoolExecutor(max_workers=15) as executor:
        futures = {
            executor.submit(get_region_summary, r): r
            for r in regions
        }
        for future in as_completed(futures):
            result = future.result()
            report.append(result)

    # Sort by instance count descending (like Global View's default)
    report.sort(key=lambda x: x.get("Instances", 0), reverse=True)
    return report

5.4 — Export Report to CSV

import csv
from datetime import datetime

def export_to_csv(report: list[dict], filename: str = None):
    if not filename:
        ts = datetime.now().strftime("%Y%m%d_%H%M%S")
        filename = f"global_view_report_{ts}.csv"

    fields = [
        "Region", "Instances", "VPCs", "Volumes",
        "SecurityGroups", "Subnets", "NatGateways",
        "InternetGateways", "ElasticIPs", "DBInstances", "Error"
    ]

    with open(filename, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
        writer.writeheader()
        writer.writerows(report)

    print(f"✅ Report exported: {filename}")
    return filename

# Run full pipeline
report = generate_global_view_report()
export_to_csv(report)

5.5 — Security Audit: Find Open Security Groups Globally

def find_open_security_groups(port: int = 22) -> list[dict]:
    """
    Scans all regions for security groups with 0.0.0.0/0 access
    on the specified port. Default: SSH (22). Common: RDP (3389).
    """
    def scan_region(region):
        ec2 = boto3.client("ec2", region_name=region)
        risky_sgs = []
        try:
            sgs = ec2.describe_security_groups(
                Filters=[
                    {"Name": "ip-permission.from-port", "Values": [str(port)]},
                    {"Name": "ip-permission.to-port", "Values": [str(port)]},
                    {"Name": "ip-permission.cidr", "Values": ["0.0.0.0/0"]},
                ]
            )["SecurityGroups"]

            for sg in sgs:
                risky_sgs.append({
                    "Region": region,
                    "GroupId": sg["GroupId"],
                    "GroupName": sg["GroupName"],
                    "VpcId": sg.get("VpcId", "EC2-Classic"),
                    "Description": sg.get("Description", ""),
                })
        except Exception:
            pass
        return risky_sgs

    regions = get_enabled_regions()
    all_risky = []
    with ThreadPoolExecutor(max_workers=15) as executor:
        results = executor.map(scan_region, regions)
        for result in results:
            all_risky.extend(result)

    print(f"⚠️  Found {len(all_risky)} security groups with port {port} open to 0.0.0.0/0")
    return all_risky

# Find SSH exposure
ssh_exposed = find_open_security_groups(port=22)
rdp_exposed = find_open_security_groups(port=3389)

5.6 — FinOps: Find Orphaned Resources Globally

def find_orphaned_resources() -> dict:
    """
    Identifies unattached EBS volumes and unassociated Elastic IPs
    across all regions — common sources of unnecessary AWS spend.
    """
    def scan_region(region):
        ec2 = boto3.client("ec2", region_name=region)
        orphans = {"region": region, "unattached_volumes": [], "unused_eips": []}
        try:
            # Unattached EBS volumes (status = available)
            vols = ec2.describe_volumes(
                Filters=[{"Name": "status", "Values": ["available"]}]
            )["Volumes"]
            orphans["unattached_volumes"] = [
                {
                    "VolumeId": v["VolumeId"],
                    "SizeGB": v["Size"],
                    "Type": v["VolumeType"],
                    "EstimatedMonthlyCost": round(v["Size"] * 0.10, 2),  # ~$0.10/GB-month gp2
                }
                for v in vols
            ]

            # Elastic IPs not associated with any instance
            eips = ec2.describe_addresses()["Addresses"]
            orphans["unused_eips"] = [
                {
                    "AllocationId": e.get("AllocationId"),
                    "PublicIp": e["PublicIp"],
                    "EstimatedMonthlyCost": 3.65,  # ~$0.005/hour idle EIP
                }
                for e in eips
                if "AssociationId" not in e
            ]
        except Exception:
            pass
        return orphans

    regions = get_enabled_regions()
    results = {}
    total_vol_cost = 0
    total_eip_cost = 0

    with ThreadPoolExecutor(max_workers=15) as executor:
        for result in executor.map(scan_region, regions):
            region = result["region"]
            if result["unattached_volumes"] or result["unused_eips"]:
                results[region] = result
                total_vol_cost += sum(
                    v["EstimatedMonthlyCost"]
                    for v in result["unattached_volumes"]
                )
                total_eip_cost += sum(
                    e["EstimatedMonthlyCost"]
                    for e in result["unused_eips"]
                )

    print(f"\n💸 Estimated monthly waste:")
    print(f"   Unattached volumes: ${total_vol_cost:.2f}/month")
    print(f"   Unused Elastic IPs: ${total_eip_cost:.2f}/month")
    print(f"   Total:              ${total_vol_cost + total_eip_cost:.2f}/month")
    return results

orphans = find_orphaned_resources()

5.7 — Resource Explorer Python Client

Using the resource-explorer-2 boto3 client to search cross-region programmatically:

import boto3

def resource_explorer_search(query: str, view_arn: str = None) -> list[dict]:
    """
    Search resources using AWS Resource Explorer v2 API.
    Requires an aggregator index in us-east-1 to be active.
    """
    client = boto3.client("resource-explorer-2", region_name="us-east-1")
    kwargs = {"QueryString": query}
    if view_arn:
        kwargs["ViewArn"] = view_arn

    resources = []
    paginator = client.get_paginator("search")
    try:
        for page in paginator.paginate(**kwargs):
            for resource in page.get("Resources", []):
                resources.append({
                    "Arn": resource["Arn"],
                    "Region": resource["Region"],
                    "ResourceType": resource["ResourceType"],
                    "Service": resource["Service"],
                    "LastReportedAt": str(resource.get("LastReportedAt", "")),
                    "Tags": {
                        tag["Key"]: tag["Value"]
                        for prop in resource.get("Properties", [])
                        if prop["Name"] == "tags"
                        for tag in prop.get("Data", [])
                    },
                })
    except client.exceptions.UnauthorizedException:
        print("❌ Resource Explorer not enabled or no aggregator index found.")
    return resources

# Search all untagged EC2 instances
untagged = resource_explorer_search(
    query="resourcetype:ec2:instance -tag.Environment"
)
print(f"Untagged EC2 instances: {len(untagged)}")

# Search production resources by tag
prod_resources = resource_explorer_search(
    query="tag.Environment=production"
)
print(f"Production-tagged resources: {len(prod_resources)}")

Section 6: Automate Global View Reports with Lambda

Deploy a daily Lambda function that generates a full Global View report and sends a cost-warning alert via SNS if waste is detected:

# lambda_function.py
import boto3
import json
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime

SNS_TOPIC_ARN = os.environ.get("SNS_TOPIC_ARN")
S3_BUCKET = os.environ.get("REPORT_BUCKET")
WASTE_THRESHOLD_USD = float(os.environ.get("WASTE_THRESHOLD", "50"))

def lambda_handler(event, context):
    regions = get_enabled_regions()
    report = []
    total_waste = 0.0

    with ThreadPoolExecutor(max_workers=15) as executor:
        futures = {
            executor.submit(get_region_summary, r): r
            for r in regions
        }
        for future in as_completed(futures):
            result = future.result()
            report.append(result)

    # Sort by instances descending
    report.sort(key=lambda x: x.get("Instances", 0), reverse=True)

    # Calculate total resources
    total_instances = sum(r.get("Instances", 0) for r in report)
    total_vpcs = sum(r.get("VPCs", 0) for r in report)
    total_volumes = sum(r.get("Volumes", 0) for r in report)

    # Save to S3
    ts = datetime.utcnow().strftime("%Y/%m/%d")
    key = f"global-view-reports/{ts}/report.json"
    s3 = boto3.client("s3")
    s3.put_object(
        Bucket=S3_BUCKET,
        Key=key,
        Body=json.dumps(report, default=str),
        ContentType="application/json",
    )
    print(f"Report saved to s3://{S3_BUCKET}/{key}")

    # SNS alert if waste exceeds threshold
    if total_waste > WASTE_THRESHOLD_USD and SNS_TOPIC_ARN:
        sns = boto3.client("sns")
        sns.publish(
            TopicArn=SNS_TOPIC_ARN,
            Subject="⚠️ AWS Global View - Waste Alert",
            Message=(
                f"Estimated monthly waste detected: ${total_waste:.2f}\n\n"
                f"Global Summary:\n"
                f"  Regions scanned: {len(regions)}\n"
                f"  Total EC2 Instances: {total_instances}\n"
                f"  Total VPCs: {total_vpcs}\n"
                f"  Total EBS Volumes: {total_volumes}\n\n"
                f"Full report: s3://{S3_BUCKET}/{key}"
            ),
        )

    return {
        "statusCode": 200,
        "regions_scanned": len(regions),
        "total_instances": total_instances,
        "total_vpcs": total_vpcs,
        "report_location": f"s3://{S3_BUCKET}/{key}",
    }

Schedule it with EventBridge:

# Create EventBridge rule — runs every day at 7 AM UTC
aws events put-rule \
  --name "GlobalViewDailyReport" \
  --schedule-expression "cron(0 7 * * ? *)" \
  --state ENABLED

# Attach Lambda target
aws events put-targets \
  --rule "GlobalViewDailyReport" \
  --targets "Id=GlobalViewLambda,Arn=arn:aws:lambda:us-east-1:123456789012:function:GlobalViewReport"

Section 7: Real-World Use Cases

Here's where AWS Global View (and its programmatic equivalents) deliver real production value:

Disaster Recovery Audit: Verify that standby EC2 instances and replicated RDS clusters exist in your DR region before a failover test
FinOps & Cost Optimization: Run the orphaned resource script monthly — unattached volumes and unused EIPs silently drain budgets in forgotten regions
Security Compliance: Use the open-SG scanner to detect SSH/RDP exposure globally; integrate into your CI/CD pipeline as a pre-deployment gate
Tag Enforcement: Use Resource Explorer's -tag.CostCenter query to find all resources missing mandatory billing tags — critical before month-end chargebacks
Region Governance: Disable opt-in regions your org never uses to reduce your attack surface and avoid accidental deployments
Migration Validation: After an AWS region migration, use the global inventory to confirm zero resources remain in the source region

Section 8: Limitations & Expert Best Practices

Limitations

Global View is read-only — pair it with AWS Config Rules for policy enforcement
It covers only EC2/VPC resource families — Lambda functions, ECS clusters, CloudFront distributions are NOT shown in Global View (use Resource Explorer for those)
Resource Explorer requires an aggregator index in us-east-1 and local indexes in all other regions you want to cover — this is not free but is very inexpensive
boto3 pagination is mandatory for accounts with >1,000 resources per region — always use paginators, never raw API calls with default limits

Expert Best Practices

Always use ThreadPoolExecutor with max_workers=15 (AWS default rate limits allow ~20 concurrent describe calls per region per account)
Cache region lists in SSM Parameter Store or DynamoDB — describe_regions adds latency and counts against API quotas when called thousands of times daily
Tag everything — Global View and Resource Explorer's tag-based search is useless without a consistent tagging strategy; enforce tags using SCP (Service Control Policies) at the org level
Combine with AWS Config — Global View tells you what exists; AWS Config tells you if it complies with your policies
Use Terraform remote state for multi-region setups — store state in S3 with DynamoDB locking, one state file per region per environment
Set AGGREGATOR index in us-east-1 — all Resource Explorer search calls must target the aggregator region; calls to local-only index regions return only local results

Conclusion

AWS Global View is one of AWS's most underutilized yet immediately actionable tools. In the console it gives you instant multi-region visibility with zero setup. Paired with the AWS CLI for scripting, Terraform and CDK for IaC, and boto3 for automation, you can build a fully programmatic, always-fresh global inventory system that does far more than what the console offers — including cost waste detection, security audits, compliance reporting, and scheduled alerting.

Kubernetes Networking & AWS EKS Networking: The Enterprise-Grade Deep Dive

Manish Kumar — Sat, 23 May 2026 16:46:37 +0000

A comprehensive reference for platform engineers, solutions architects, and DevOps teams building production-grade container infrastructure.

1. Why Kubernetes Networking Is Hard

Kubernetes treats the network as a first-class citizen. Every design decision—from how two containers share a loopback interface, to how global traffic lands on a specific pod replica—flows from four fundamental problems that Kubernetes explicitly solves:

Container-to-container communication inside a Pod (solved via shared network namespace / localhost)
Pod-to-Pod communication across nodes (solved by the CNI plugin)
Pod-to-Service communication (solved by kube-proxy and iptables/eBPF rules)
External-to-Service communication (solved by Ingress, LoadBalancer services, and Gateway API)

Understanding these four planes separately—then understanding how they interact on AWS EKS—is the difference between a cluster that merely runs and one that is observable, secure, and cost-efficient at enterprise scale.

2. The Kubernetes Networking Model: Core Contracts

The Kubernetes networking model imposes three non-negotiable rules that every CNI implementation must satisfy:

Every Pod gets its own routable IP address
All Pods can communicate with all other Pods without NAT
Agents on a node (kubelet, kube-proxy) can communicate with all Pods on that node

These rules mean Kubernetes treats Pods like VMs: you can always reach any Pod directly by its IP. The challenge is implementing this across a multi-node cluster with thousands of ephemeral processes—which is exactly what CNI plugins handle.

3. CNI: The Container Network Interface

What CNI Does

CNI (Container Network Interface) is the standard plugin API that the container runtime (containerd, CRI-O) calls when a Pod is created or deleted. The runtime executes the CNI binary, which is responsible for:

Attaching a virtual network interface to the Pod's network namespace
Assigning an IP address (via an IPAM sub-plugin)
Configuring routing so the Pod can talk to the outside world

Popular CNI Plugins at Enterprise Scale

Plugin	Data Plane	Network Policy	Key Strength
AWS VPC CNI	Native AWS ENI	✅ (since 2023)	Direct VPC routing, no overlay
Calico	iptables / eBPF	✅	Advanced policy, BGP peering
Cilium	eBPF	✅	Highest performance, observability
Flannel	VXLAN overlay	❌	Simple, low overhead
Weave Net	VXLAN	✅	Easy multi-cloud

For EKS, AWS VPC CNI is the default and the AWS-supported path. For workloads requiring advanced network policy (microsegmentation, L7 policies, or service mesh), Cilium is the recommended alternative.

CNI Installation (Helm – Cilium example)

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=<API_SERVER_ENDPOINT> \
  --set k8sServicePort=443 \
  --set ipam.mode=kubernetes \
  --set nodeinit.enabled=true

4. Pod Networking Internals

The veth Pair Model

When Kubernetes creates a Pod, the container runtime creates a veth (virtual Ethernet) pair: one end sits in the Pod's network namespace (eth0 from the Pod's perspective), and the other end sits in the root namespace of the node, attached to the node's bridge (cbr0 or cni0). Traffic flows:

Pod eth0  ←→  veth pair  ←→  node bridge  ←→  routing table  ←→  other pod

Checking Pod Network Namespace (kubectl + nsenter)

# Get the pod's node and container ID
kubectl get pod nginx-pod -o wide
kubectl get pod nginx-pod \
  -o jsonpath='{.status.containerStatuses[0].containerID}'

# On the node, inspect the network namespace
PID=$(crictl inspect <CONTAINER_ID> | jq '.info.pid')
nsenter -t $PID -n ip addr
nsenter -t $PID -n ip route

# Verify connectivity between pods
kubectl exec -it nginx-pod -- curl http://<other-pod-ip>:8080

Overlay vs Underlay Networks

Overlay (VXLAN/Geneve): Pod packets are encapsulated inside UDP and decapsulated on the destination node. Works anywhere but adds ~10–15% overhead.
Underlay (BGP/Native): Pod IPs are advertised directly to the physical routing fabric. Zero overhead but requires router configuration.
AWS VPC CNI: Uses underlay—Pod IPs are native VPC IPs assigned directly to EC2 ENIs. No encapsulation, no overhead.

SNAT and Why It Matters

When a Pod communicates with an address outside the cluster, the node performs Source NAT (SNAT), replacing the Pod IP with the Node IP. This keeps the connection routable. AWS VPC CNI avoids this for intra-VPC traffic because Pod IPs are already routable within the VPC.

5. Kubernetes Services: The Stable Networking Layer

Pods are ephemeral. A Service provides a stable virtual IP (ClusterIP) that front-ends a dynamic set of Pod replicas. kube-proxy watches the API server and programs iptables (or ipvs) rules so that any traffic to the ClusterIP is DNAT-ed to a healthy Pod endpoint.

Service Types

# ClusterIP – internal only
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
  namespace: production
spec:
  type: ClusterIP
  selector:
    app: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

---
# NodePort – opens a port on every node
apiVersion: v1
kind: Service
metadata:
  name: backend-nodeport
spec:
  type: NodePort
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080   # Range: 30000-32767

---
# LoadBalancer – provisions an external load balancer (NLB/ALB on EKS)
apiVersion: v1
kind: Service
metadata:
  name: backend-lb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
spec:
  type: LoadBalancer
  selector:
    app: backend
  ports:
    - port: 443
      targetPort: 8443

Headless Services (for StatefulSets / DNS-based discovery)

apiVersion: v1
kind: Service
metadata:
  name: cassandra
spec:
  clusterIP: None      # Headless: no VIP, DNS returns pod IPs directly
  selector:
    app: cassandra
  ports:
    - port: 9042

kube-proxy Modes

# Check current kube-proxy mode on EKS
kubectl get configmap kube-proxy-config -n kube-system -o yaml | grep mode

# Switch to IPVS mode (more scalable, O(1) service lookup)
kubectl edit configmap kube-proxy-config -n kube-system
# Change: mode: "iptables" → mode: "ipvs"

At scale (1000+ Services), IPVS dramatically outperforms iptables. With Cilium's kubeProxyReplacement=strict, kube-proxy is eliminated entirely in favor of eBPF.

6. DNS in Kubernetes (CoreDNS)

CoreDNS is the cluster DNS server. Every Pod gets /etc/resolv.conf pointing to the CoreDNS ClusterIP. Service discovery follows this FQDN pattern:

<service-name>.<namespace>.svc.cluster.local

# Test DNS resolution from a debug pod
kubectl run debug --image=busybox --rm -it -- nslookup backend-svc.production.svc.cluster.local

# Inspect CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml

# Scale CoreDNS for high-traffic clusters
kubectl scale deployment coredns -n kube-system --replicas=4

CoreDNS Custom ConfigMap (add stubzone for internal domain)

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health { lameduck 5s }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
    internal.corp:53 {
        forward . 10.0.0.2
    }
}

7. Network Policies: Microsegmentation

By default, all Pods can communicate with all other Pods—a zero-trust nightmare in multi-tenant environments. NetworkPolicy resources define allow-list rules at the L3/L4 level.

Default Deny All (Zero-Trust Baseline)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}      # Selects ALL pods in namespace
  policyTypes:
    - Ingress
    - Egress

Allow Frontend → Backend on Port 8080

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Allow Egress to AWS RDS Only

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-db-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.20.0.0/24   # RDS subnet CIDR
      ports:
        - protocol: TCP
          port: 3306
    - to: []                       # Allow DNS
      ports:
        - protocol: UDP
          port: 53

Important: Network policies are enforced by the CNI plugin, not Kubernetes core. AWS VPC CNI supports Kubernetes NetworkPolicy API natively since August 2023. For L7 policies (HTTP path/method-based), you need Cilium or a service mesh.

8. Ingress and Gateway API

NGINX Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.replicaCount=3

Ingress Resource (TLS termination)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls-secret
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /v1
            pathType: Prefix
            backend:
              service:
                name: api-svc
                port:
                  number: 80
          - path: /static
            pathType: Prefix
            backend:
              service:
                name: static-svc
                port:
                  number: 80

Gateway API (Next-Gen Ingress)

Gateway API is the successor to Ingress, offering richer routing semantics and role-based separation (infrastructure vs. application team concerns):

# GatewayClass – provisioned by cluster admin
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: aws-gateway
spec:
  controllerName: eks.amazonaws.com/gateway-controller

---
# Gateway – owned by network team
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
  namespace: production
spec:
  gatewayClassName: aws-gateway
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: prod-tls-cert

---
# HTTPRoute – owned by app team
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-route
  namespace: production
spec:
  parentRefs:
    - name: prod-gateway
  hostnames:
    - "api.example.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: api-svc
          port: 80
          weight: 100

9. Service Mesh (Istio / AWS App Mesh)

At enterprise scale, L4 Network Policies are insufficient. A service mesh inserts a sidecar proxy (Envoy) into each Pod, giving you:

mTLS between all services (zero-trust encryption)
Traffic shaping: canary, blue/green, weighted routing
Observability: distributed tracing, latency histograms per-route
L7 policies: rate limiting, circuit breaking, retry logic

# Install Istio on EKS
curl -L https://istio.io/downloadIstio | sh -
istioctl install --set profile=production -y

# Enable automatic sidecar injection for a namespace
kubectl label namespace production istio-injection=enabled

# Verify mesh
kubectl get pods -n production   # Each pod should show 2/2 READY (app + envoy)
istioctl analyze -n production

Istio VirtualService (Canary Routing 90/10)

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: backend-vs
  namespace: production
spec:
  hosts:
    - backend-svc
  http:
    - route:
        - destination:
            host: backend-svc
            subset: v1
          weight: 90
        - destination:
            host: backend-svc
            subset: v2
          weight: 10

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: backend-dr
  namespace: production
spec:
  host: backend-svc
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL    # Enforce mTLS
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

10. AWS EKS Networking Architecture

VPC Design for EKS

Enterprise EKS deployments require a carefully planned VPC:

VPC: 10.0.0.0/16
├── Public Subnets (one per AZ) – NAT Gateways, Load Balancers
│   ├── 10.0.1.0/24  (us-east-1a)
│   ├── 10.0.2.0/24  (us-east-1b)
│   └── 10.0.3.0/24  (us-east-1c)
├── Private Subnets (Worker Nodes)
│   ├── 10.0.11.0/24 (us-east-1a)
│   ├── 10.0.12.0/24 (us-east-1b)
│   └── 10.0.13.0/24 (us-east-1c)
└── Pod Subnets (large /19 blocks for IP density)
    ├── 10.0.128.0/19 (us-east-1a)
    ├── 10.0.160.0/19 (us-east-1b)
    └── 10.0.192.0/19 (us-east-1c)

Subnet tagging is mandatory for the AWS Load Balancer Controller to auto-discover subnets:

# Tag public subnets for external ALB/NLB
aws ec2 create-tags --resources subnet-xxxxxx \
  --tags Key=kubernetes.io/role/elb,Value=1 \
         Key=kubernetes.io/cluster/<cluster-name>,Value=shared

# Tag private subnets for internal LBs
aws ec2 create-tags --resources subnet-yyyyyy \
  --tags Key=kubernetes.io/role/internal-elb,Value=1 \
         Key=kubernetes.io/cluster/<cluster-name>,Value=shared

11. AWS VPC CNI: How It Works

AWS VPC CNI is the default—and most performant—CNI for EKS. Each Pod gets a native VPC IP address allocated from a secondary IP address pool on the node's ENI. There is no overlay network; Pod traffic is routed directly in the VPC fabric.

The ENI/IP Math

Each EC2 instance type supports a maximum number of ENIs and secondary IPs per ENI. The formula for maximum Pod density per node is:

Max Pods = (Max ENIs × (IPs per ENI - 1)) + 2

For m5.large (3 ENIs × (10 IPs per ENI - 1) + 2 = 27 + 2 = 29 pods max).

# Check current VPC CNI version
kubectl describe daemonset aws-node -n kube-system | grep Image

# Check IP allocation on a node
kubectl get node ip-10-0-11-50.ec2.internal \
  -o jsonpath='{.metadata.annotations.vpc\.amazonaws\.com/node-capacity}'

# Enable prefix delegation (multiplies IP capacity by 16)
kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true \
  WARM_PREFIX_TARGET=1

Custom Networking (Separate Pod CIDR)

When node and Pod IPs must live in different subnets—common when IPv4 space is scarce:

# Step 1: Enable custom networking
kubectl set env daemonset aws-node -n kube-system \
  AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true \
  ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

# Step 2: Create ENIConfig per AZ
cat <<EOF | kubectl apply -f -
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-east-1a
spec:
  subnet: subnet-pod-subnet-1a-id      # Pod-dedicated subnet
  securityGroups:
    - sg-xxxxxxxxxxxxxxxxx
EOF

12. Provisioning EKS with Terraform (Full Networking Stack)

# terraform/main.tf

terraform {
  required_providers {
    aws        = { source = "hashicorp/aws", version = "~> 5.0" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.0" }
    helm       = { source = "hashicorp/helm", version = "~> 2.0" }
  }
}

provider "aws" {
  region = var.region
}

# ── VPC ────────────────────────────────────────────────────────────────────
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.cluster_name}-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.11.0/24", "10.0.12.0/24", "10.0.13.0/24"]
  public_subnets  = ["10.0.1.0/24",  "10.0.2.0/24",  "10.0.3.0/24"]
  intra_subnets   = ["10.0.128.0/19","10.0.160.0/19","10.0.192.0/19"] # Pod subnets

  enable_nat_gateway     = true
  single_nat_gateway     = false   # One per AZ for HA
  enable_vpn_gateway     = false
  enable_dns_hostnames   = true
  enable_dns_support     = true

  public_subnet_tags = {
    "kubernetes.io/role/elb"                        = "1"
    "kubernetes.io/cluster/${var.cluster_name}"     = "shared"
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"               = "1"
    "kubernetes.io/cluster/${var.cluster_name}"     = "shared"
  }
}

# ── EKS Cluster ───────────────────────────────────────────────────────────
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.30"

  cluster_endpoint_public_access       = true
  cluster_endpoint_private_access      = true
  cluster_endpoint_public_access_cidrs = ["YOUR_BASTION_IP/32"]   # Restrict API access

  vpc_id                   = module.vpc.vpc_id
  subnet_ids               = module.vpc.private_subnets
  control_plane_subnet_ids = module.vpc.intra_subnets

  # EKS Addons
  cluster_addons = {
    coredns = {
      most_recent = true
      configuration_values = jsonencode({
        replicaCount = 3
        resources = {
          limits   = { cpu = "200m", memory = "256Mi" }
          requests = { cpu = "100m", memory = "128Mi" }
        }
      })
    }
    kube-proxy = { most_recent = true }
    vpc-cni = {
      most_recent = true
      configuration_values = jsonencode({
        env = {
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
        }
      })
    }
    aws-ebs-csi-driver = { most_recent = true }
  }

  # Managed Node Groups
  eks_managed_node_groups = {
    system = {
      name           = "system-ng"
      instance_types = ["m5.xlarge"]
      min_size       = 2
      max_size       = 4
      desired_size   = 3

      subnet_ids = module.vpc.private_subnets

      labels = { role = "system" }
      taints = [{
        key    = "CriticalAddonsOnly"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]
    }

    application = {
      name           = "app-ng"
      instance_types = ["m5.2xlarge", "m5.4xlarge"]
      capacity_type  = "SPOT"
      min_size       = 3
      max_size       = 30
      desired_size   = 6

      subnet_ids = module.vpc.private_subnets

      labels = { role = "application" }
    }
  }

  # Security Groups for Pods (Security Groups for Pods feature)
  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["0.0.0.0/0"]
    }
  }
}

# ── AWS Load Balancer Controller ──────────────────────────────────────────
module "aws_lbc_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.0"

  role_name                              = "${var.cluster_name}-aws-lbc"
  attach_load_balancer_controller_policy = true

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:aws-load-balancer-controller"]
    }
  }
}

resource "helm_release" "aws_lbc" {
  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  namespace  = "kube-system"
  version    = "1.8.0"

  set { name = "clusterName";                     value = var.cluster_name }
  set { name = "serviceAccount.create";           value = "true" }
  set { name = "serviceAccount.name";             value = "aws-load-balancer-controller" }
  set { name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
        value = module.aws_lbc_irsa.iam_role_arn }
  set { name = "replicaCount";                    value = "2" }
  set { name = "vpcId";                           value = module.vpc.vpc_id }
  set { name = "region";                          value = var.region }
}

13. AWS Load Balancer Controller: ALB Ingress

The AWS Load Balancer Controller integrates natively with ALB and NLB, replacing the need for NGINX in many EKS workloads.

# ALB Ingress with HTTPS, WAF, and target type IP
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: production-ingress
  namespace: production
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip          # Route to Pod IPs directly
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT_ID
    alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:us-east-1:ACCOUNT:regional/webacl/NAME/ID
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/success-codes: "200,201"
    alb.ingress.kubernetes.io/load-balancer-attributes: >-
      idle_timeout.timeout_seconds=120,
      routing.http2.enabled=true,
      access_logs.s3.enabled=true,
      access_logs.s3.bucket=my-alb-logs,
      access_logs.s3.prefix=eks-prod
spec:
  ingressClassName: alb
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-svc
                port:
                  number: 80

14. Security Groups for Pods (SGP)

A powerful EKS-specific feature: assign AWS Security Groups directly to individual Pods (not just nodes), enabling fine-grained access to AWS services like RDS, ElastiCache, and MSK.

# Enable Security Groups for Pods on VPC CNI
kubectl set env daemonset aws-node -n kube-system \
  ENABLE_POD_ENI=true

# Verify trunk ENI support on nodes
kubectl describe node <node-name> | grep vpc.amazonaws.com/has-trunk-attached

# SecurityGroupPolicy CRD – assign SG to specific pods
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: backend-sgp
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  securityGroups:
    groupIds:
      - sg-backend-to-rds-id       # SG that allows 3306 to RDS
      - sg-backend-to-redis-id     # SG that allows 6379 to ElastiCache

15. EKS Cluster Networking: CLI Reference

# ── Cluster Creation with eksctl ──────────────────────────────────────────
cat <<EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod-cluster
  region: us-east-1
  version: "1.30"

vpc:
  cidr: 10.0.0.0/16
  clusterEndpoints:
    publicAccess:  true
    privateAccess: true
  publicAccessCIDRs: ["YOUR_IP/32"]

addons:
  - name: vpc-cni
    version: latest
    configurationValues: |
      {"env":{"ENABLE_PREFIX_DELEGATION":"true","WARM_PREFIX_TARGET":"1"}}
  - name: coredns
    version: latest
  - name: kube-proxy
    version: latest
  - name: aws-ebs-csi-driver
    version: latest

managedNodeGroups:
  - name: system
    instanceType: m5.xlarge
    desiredCapacity: 3
    minSize: 2
    maxSize: 6
    privateNetworking: true
    subnets:
      - private-us-east-1a
      - private-us-east-1b
      - private-us-east-1c
    taints:
      - key: CriticalAddonsOnly
        value: "true"
        effect: NoSchedule

  - name: application
    instanceTypes: ["m5.2xlarge", "m5.4xlarge"]
    desiredCapacity: 6
    minSize: 3
    maxSize: 30
    privateNetworking: true
    spot: true
EOF

eksctl create cluster -f cluster.yaml

# ── Network Diagnostics ───────────────────────────────────────────────────
# Check all node IPs and ENI capacity
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,IP:.status.addresses.address,\
MAX-PODS:.status.allocatable.pods'

# Describe VPC CNI daemonset
kubectl describe daemonset aws-node -n kube-system

# Check CNI logs for IP allocation issues
kubectl logs -n kube-system -l k8s-app=aws-node --tail=100

# Verify pod networking end-to-end
kubectl run nettest --image=nicolaka/netshoot --rm -it -- \
  traceroute 10.0.11.100     # Trace route to another pod IP

# Check network policies in effect
kubectl get networkpolicies -A
kubectl describe networkpolicy default-deny-all -n production

16. Observability: Network Monitoring at Enterprise Scale

CloudWatch Container Insights (EKS)

# Install CloudWatch agent via Helm
helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm install aws-cloudwatch-metrics aws-observability/aws-cloudwatch-metrics \
  --namespace amazon-cloudwatch --create-namespace \
  --set clusterName=prod-cluster

VPC Flow Logs (mandatory for compliance)

# In Terraform – enable VPC flow logs to S3
resource "aws_flow_log" "eks_vpc" {
  vpc_id          = module.vpc.vpc_id
  traffic_type    = "ALL"
  iam_role_arn    = aws_iam_role.flow_logs.arn
  log_destination = aws_s3_bucket.flow_logs.arn
  log_destination_type = "s3"

  log_format = "$${version} $${account-id} $${vpc-id} $${subnet-id} $${srcaddr} $${dstaddr} $${srcport} $${dstport} $${protocol} $${packets} $${bytes} $${action} $${log-status}"
}

Cilium Hubble (eBPF-based network observability)

# Enable Hubble UI
helm upgrade cilium cilium/cilium -n kube-system \
  --reuse-values \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# Port-forward Hubble UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80 &
# Open http://localhost:12000 for visual flow maps

# CLI: watch live network flows
hubble observe --namespace production --follow

17. IPv6 and Dual-Stack on EKS

AWS EKS supports IPv6-only clusters, which eliminates IPv4 exhaustion issues. In IPv6 mode, each Pod gets a unique IPv6 address from the subnet's /80 CIDR allocation (the VPC is assigned a /56, which is divided into /64 subnets, and pods receive addresses from a /80 prefix per node).

# Create IPv6 EKS cluster with eksctl
cat <<EOF | eksctl create cluster -f -
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ipv6-cluster
  region: us-east-1
  version: "1.30"
kubernetesNetworkConfig:
  ipFamily: IPv6
managedNodeGroups:
  - name: ng-ipv6
    instanceType: m5.xlarge
    desiredCapacity: 3
EOF

18. Enterprise Networking Checklist

Before a production EKS cluster is considered network-hardened, verify the following:

[ ] VPC design — Dedicated pod subnets, no IP overlap with on-prem, NAT per AZ
[ ] Subnet tagging — Public and private subnets tagged for LB auto-discovery
[ ] VPC CNI — Prefix delegation enabled, custom networking for IP separation
[ ] Endpoint access — Public API server restricted to bastion/VPN CIDRs; private access enabled
[ ] Security Groups for Pods — Configured for workloads accessing AWS managed services
[ ] Default-deny NetworkPolicy — Applied to all production namespaces
[ ] CoreDNS — Scaled to ≥2 replicas; custom stub zones for internal DNS
[ ] AWS LBC — Deployed with IRSA; target-type: ip for ALB
[ ] VPC Flow Logs — Enabled for ALL traffic to S3 with Athena queries configured
[ ] Service Mesh — mTLS enforced for internal service-to-service communication
[ ] kube-proxy mode — IPVS for clusters with >100 services; Cilium kube-proxy replacement for >500
[ ] Security Group NACLs — Stateless NACL rules for defense-in-depth on cluster subnets
[ ] Prefix delegation — Enabled for clusters expecting >29 pods/node
[ ] IPv6 readiness — Evaluated for new cluster deployments to future-proof IP management

19. Troubleshooting Cookbook

# Pod can't reach external internet
kubectl exec -it <pod> -- curl -v https://example.com
# Check: NAT Gateway route in private subnet route table

# DNS resolution failing
kubectl exec -it <pod> -- nslookup kubernetes.default
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Check: CoreDNS pod health, security group allows UDP/TCP 53 from nodes

# Service not reachable
kubectl get endpoints <service-name> -n <namespace>
# If endpoints list is empty: label selector on Service doesn't match pod labels

# ALB not provisioned
kubectl describe ingress <ingress-name> -n <namespace>
kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller
# Check: subnet tags, IRSA role permissions, security group for ALB

# IP exhaustion on nodes
kubectl describe node <node> | grep -A5 "Allocated resources"
# Fix: enable prefix delegation or add nodes with higher ENI capacity

# NetworkPolicy blocking traffic unexpectedly
kubectl get networkpolicies -n <namespace>
# Use: kubectl exec debug pod and curl; use Cilium Hubble or "kubectl-netpol" for visualization

# Pods stuck in ContainerCreating (CNI error)
kubectl describe pod <pod-name>
kubectl logs -n kube-system aws-node-<node-id>
# Common: WARM_IP_TARGET exhausted, ENI attach limit reached

Kubernetes networking is an architectural discipline, not just a configuration task. At enterprise scale on AWS EKS, the combination of VPC CNI for native performance, Security Groups for Pods for fine-grained AWS service access, the AWS Load Balancer Controller for cloud-native ingress, NetworkPolicy for microsegmentation, and a service mesh for mTLS and L7 observability creates a security-hardened, observable, and cost-efficient networking stack that can scale from dozens to thousands of pods without redesign.

Cross‑Account Database Migration with AWS DMS: A Production‑Ready Architecture

Manish Kumar — Mon, 18 May 2026 03:56:56 +0000

This blog presents an objective, engineer‑first view of cross‑account AWS DMS setups for multi‑account AWS environments. It covers:

Why do you need cross‑account DMS?
Connectivity patterns: VPC peering, public‑internet, Transit Gateway, and PrivateLink.
Terraform‑based, multi‑account IaC with provider aliases and workspaces.
Validation, monitoring, alerts, costs, and common DMS limits.

1. Why Cross‑Account DMS Is Needed

Organizations that run multiple AWS accounts often keep source and target databases in different accounts for reasons such as:

Security and governance (least‑privilege, separation of duties).
Billing and charge‑back (each team owns its own account).
Compliance (ISO, SOC, PCI environments strictly isolated).
Mergers and migrations (green‑field target account vs legacy source account).

AWS DMS itself does not care about the account boundary; all it requires is:

Network reachability from the DMS replication instance in one account to the source and target endpoints.
Enough IAM permissions and security‑group rules to read and write data.

The core challenge is therefore networking and permissions, not DMS configuration.

2. Connectivity Patterns Overview

For cross‑account DMS, you typically choose one of four patterns:

VPC peering between the DMS‑account VPC and the source‑DB‑account VPC.
Public‑internet‑based with tight IP allowlists.
Transit Gateway‑based routing across many accounts.
PrivateLink (if using services that expose endpoint‑services).

This blog focuses first on VPC peering and public‑internet patterns, then expands on Transit Gateway and PrivateLink.

3. Pattern 1 — VPC Peering

3.1 Architecture

Account A (DMS account): hosts the DMS replication instance in a private VPC.
Account B (Source account): hosts the source database (for example, Aurora MySQL) in its own VPC.
A VPC peering connection links the two VPCs.
Route tables in both VPCs route each other’s CIDRs via the peering connection.
Security groups on the database allow inbound traffic from the DMS VPC CIDR or the DMS security‑group ID.

All traffic stays inside AWS’s backbone; no public IP exposure is required.

3.2 Console Steps

Step 1: Create VPC peering (Account A)

VPC → Peering Connections → Create Peering Connection.
Name: dms‑src‑db‑peering.
VPC Requester: VPC where DMS runs.
VPC accepter: select Another account and enter Account B’s account ID plus the VPC ID of the source DB.

Step 2: Accept peering (Account B)

In Account B’s VPC console, accept the pending peering request.
Wait for status Active.

Step 3: Update route tables

In Account A, add a route to the source‑DB VPC CIDR via the peering connection.
In Account B, add a route to the DMS VPC CIDR via the same peering connection.

Step 4: Configure security groups (Account B)

In the DB’s security group, add an inbound rule for the database port (3306, 5432, 5439, etc.) with the DMS VPC CIDR as the source.
In Account A, ensure the DMS security group permits outbound traffic to the DB VPC CIDR.

Step 5: Create DMS resources (Account A)

Create a DMS replication instance in the DMS VPC.
Create Source and Target endpoints.
Create a Replication task with full‑load‑and‑cdc or one‑time full‑load.

4. Pattern 2 — Public Internet

If VPC peering is not allowed (CIDR overlap, governance, or cross‑account restriction), you can still use DMS over the public internet, provided you tighten the security model.

4.1 How it works

The DMS replication instance in Account A is in a public or NAT‑gateway‑backed subnet.
The source DB in Account B allows inbound traffic only from the DMS instance’s public IP or the NAT gateway’s public IP.
Optionally, an SNS + Lambda stack in Account B updates the DB security group automatically when the DMS instance IP changes.

4.2 Security‑group automation example

A typical pattern:

SNS topic in Account A publishes replication‑instance‑creation events.
Lambda function in Account B is subscribed to that topic.
Lambda:
- Calls dms:describe‑replication‑instances to get the public IP.
- Calls ec2:authorize‑security‑group‑ingress and revoke‑security‑group‑ingress on the DB security group.

5. Transit Gateway Option

5.1 When Transit Gateway Is Better Than Peering

Transit Gateway is preferable to VPC peering in large, multi‑account environments because:

It avoids an N‑squared mesh of VPC peers.
Centralized routing and policy enforcement simplify compliance and change control.
You can integrate VPN, Direct Connect, and multiple accounts through a single hub.

Tradeoffs vs peering:

Peering is simpler for one‑off or few‑account links but does not scale.
Transit Gateway scales well but adds cost and routing‑table complexity.

5.2 Scaling and Route Management

In a Transit Gateway‑based setup:

Each account’s VPC is attached to the Transit Gateway (TGW).
TGW route tables define which VPCs can talk to which others (for example, “DMS VPCs” → “DB VPCs”).
You can use route tables with tags (for example, Environment = Prod) to control connectivity at scale rather than per‑VPC.

Each additional VPC attachment typically costs per‑hour and per‑GB of data processed through the TGW, so over‑provisioning attachments or allowing unnecessary traffic can increase cost quickly.

5.3 Overlapping VPCs and Cross‑Account Complexity

CIDR overlaps are still a problem: Transit Gateway will not allow overlapping CIDRs in the same routing domain.
Best practice: assign non‑overlapping CIDR ranges at the organization level (for example, 10.0.x.0/24 per account).
When multiple teams share a Transit Gateway, ensure resource‑based permissions and tagging are used to prevent unauthorized VPC attachments.

6. PrivateLink Option

PrivateLink is useful when you want to expose the source or target database as a VPC endpoint service and let DMS reach it without routing through the public internet or a VPC‑peering mesh.

The database runs in a Network Load Balancer‑exposed VPC endpoint service.
DMS VPC creates a VPC endpoint pointing to that service.
Traffic is private, and the NLB can be in a different account than the DMS instance.

PrivateLink is ideal for SaaS‑style databases or centrally managed data platforms, but it adds cost and operational complexity compared with direct VPC peering or Transit Gateway.

7. Terraform Implementation (Enterprise Multi‑Account Pattern)

7.1 Multi‑account providers

For cross‑account DMS, configure separate AWS providers:

# main.tf
provider "aws" {
  alias = "account_a"

  region = "us-east-1"
  profile = "account-a"
}

provider "aws" {
  alias = "account_b"

  region = "us-east-1"
  profile = "account-b"
}

Then, in account‑specific modules:

# account_a/dms.tf
resource "aws_dms_replication_instance" "cross_account" {
  provider = aws.account_a

  replication_instance_identifier = "dms-replication-instance-cross-account"
  replication_instance_class      = "dms.t3.medium"
  allocated_storage               = 50
  vpc_security_group_ids          = [aws_security_group.dms_sg.id]
  replication_subnet_group_id     = aws_dms_replication_subnet_group.dms_subnet.id
}

resource "aws_dms_endpoint" "src_db" {
  provider = aws.account_a

  endpoint_identifier = "src-db"
  endpoint_type       = "source"
  engine_name         = "mysql"
  server_name         = "aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com"
  port                = 3306
  username            = "myuser"
  password            = "mypassword"
}

resource "aws_dms_replication_task" "cross_account_task" {
  provider = aws.account_a

  replication_task_identifier = "cross-account-migration-task"
  replication_instance_arn    = aws_dms_replication_instance.cross_account.replication_instance_arn
  source_endpoint_arn         = aws_dms_endpoint.src_db.endpoint_arn
  target_endpoint_arn         = aws_dms_endpoint.tgt_redshift.endpoint_arn
  migration_type              = "full-load-and-cdc"
  table_mappings              = jsonencode({
    rules = [
      {
        rule-type  = "selection"
        rule-id    = "1"
        rule-name  = "1"
        object-locator = {
          schema-name = "%"
          table-name  = "%"
        }
        rule-action = "include"
      }
    ]
  })
}

7.2 Remote state and workspaces

Use remote state (for example, terraform { backend "s3" { ... } }) to coordinate changes between Account A and Account B.
Use Terraform workspaces (or environment‑specific stacks) to separate prod and staging without duplicating configuration.
In CI/CD pipelines, run plan and apply separately for each account, checking that VPC peering, TGW attachments, and security‑group rules are aligned.

8. Troubleshooting

This section assumes you have already reviewed the core connectivity and IAM steps above.

8.1 Basic prereqs

Before deep‑troubleshooting:

Confirm DMS replication instance is available.
Confirm source and target databases are reachable from an EC2 test instance in the same VPC.
Check CloudWatch DMS log groups for ERR_‑prefixed messages.

Console:
DMS → Replication instances → select instance → Events tab.

CLI:

aws dms describe-replication-instances \
  --replication-instance-identifier dms-replication-instance-cross-account \
  --profile account-a

8.2 Validation and connectivity commands

From a test instance in the DMS‑account VPC (or the DMS instance itself if allowed):

TCP connectivity to MySQL/Aurora:

nc -zv aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com 3306

PostgreSQL:

nc -zv pg-source.cluster-xyz.us-east-1.rds.amazonaws.com 5432

DNS resolution:

nslookup aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com
dig aurora-cluster.cluster-xyz.us-east-1.rds.amazonaws.com

If nc or telnet fail, but nslookup succeeds, the problem is likely security‑group or route‑table, not DNS.

8.3 Security‑group and VPC checks (CLI)

Security group rules (Account B):

aws ec2 describe-security-groups \
  --group-ids sg-0x1y2z3w4v5u6t7s8 \
  --profile account-b

Route tables (Accounts A and B):

aws ec2 describe-route-tables \
  --route-table-ids rtb-0a1b2c3d4e5f6g7h8 \
  --query 'RouteTables.Routes' \
  --profile account-a

Public IP of DMS instance or NAT gateway (Account A):

aws ec2 describe-network-interfaces \
  --filters "Name=description,Values=*DMS Replication Instance*" \
  --query 'NetworkInterfaces.Association.PublicIp' \
  --profile account-a

If the public IP changes, update the DB‑side security group accordingly or use Lambda automation.

8.4 IAM and DMS‑specific issues

Symptoms:

Endpoint test succeeds, but task fails to start.
Errors like Premigration assessment failed or Can’t start replication task.

Check:

IAM role attached to the DMS replication instance has permissions for source and target engines.
Cross‑account Secrets Manager or S3 policies allow the DMS role from Account A to read the secret or S3 bucket in Account B.

CLI:

aws iam get-role \
  --role-name dms-replication-instance-role \
  --profile account-a

Also check CloudWatch logs for ERR_ messages and inspect terraform plan to ensure the DMS IAM role is not accidentally mismatched across accounts.

9. Monitoring & Alerting

9.1 CloudWatch metrics to watch

For each DMS task, monitor:

Metric	What it tells you
`CDCLatencySource` / `CDCLatencyTarget`	Seconds between change on source and target .
`FullLoadRows` / `FullLoadDuration`	Progress of full‑load phase.
`StorageConsumption`	How much storage your task uses on the replication instance.
`CPUUtilization`	CPU load on the DMS instance.
`FreeStorageSpace`	Bytes of storage remaining.

You can view these in the DMS console (Tasks → select task → Monitoring table) or via CloudWatch.

9.2 CloudWatch alarms

Create alarms for:

High CDC latency (for example, above 30 seconds for realtime‑style workloads).
High CPUUtilization (for example, >70% for prolonged periods).
Low FreeStorageSpace (for example, <10 GB).
Task status change to Failed or Stopped.

Example CLI alarm for CDC latency:

aws cloudwatch put-metric-alarm \
  --alarm-name "dms-cross-account-cdc-latency-high" \
  --alarm-description "CDC latency too high on cross‑account migration task" \
  --metric-name OverallCDCLatency \
  --namespace AWS/DMS \
  --statistic Average \
  --period 300 \
  --threshold 30 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:dms-alerts \
  --profile account-a

9.3 EventBridge notifications

Configure EventBridge rules to capture DMS‑related events:

DMS replication task state changes.
DMS instance state changes.

This lets you send notifications to SNS, SQS, or Lambda for automated remediation or escalation.

10. Production Best Practices

Minimize DMS‑instance lifetime: spin up the replication instance only for the migration window; delete it afterward to save cost.
Pre‑test connectivity and permissions from a test EC2 instance before creating DMS resources.
Use VPC peering or TGW instead of public‑internet for CDC‑style workloads.
Automate security‑group updates with Lambda+SNS for IP‑based patterns.
Validate IAM roles and policies across accounts early in the pipeline.

11. Cost Considerations

11.1 DMS instance pricing

AWS DMS charges per hour for replication instances (for example, dms.t3.medium, dms.c5.large, etc.), plus storage and, in some cases, per‑hour fees for serverless variants.

On‑demand: price per hour scales with compute and storage (from roughly a few cents per hour for t3.micro to several dollars per hour for r6i.32xlarge‑class instances).
Serverless: billed per‑hour‑of‑usage plus per‑GB of data‑stored, which can be attractive for short‑lived migrations but can become costly for long‑running CDC.

Best practice: size the DMS instance based on the volume of changes per second and expected migration window, then shrink or stop it once migration completes.

11.2 NAT Gateway costs

If the DMS instance is in a private subnet, it typically uses a NAT gateway for outbound internet traffic:

NAT Gateway pricing: hourly charge plus per‑GB egress data.
For large migrations, egress traffic through the NAT gateway can add up, especially if the DMS instance also talks to public S3 buckets or Secrets Manager endpoints.

To reduce cost, use VPC endpoints (for S3, Secrets Manager, etc.) instead of NAT‑gateway‑based public‑internet traffic.

11.3 Transit Gateway vs VPC peering costs

VPC peering has no direct hourly charge, but it does not scale to many accounts and requires manual route‑table maintenance.
Transit Gateway:
- Hourly charge per TGW and per attachment (for example, about \$0.05/hour per VPC attachment in some regions).
- Data‑processing charge per GB sent through the TGW (for example, \$0.02/GB).
For a handful of cross‑account links, VPC peering is usually cheaper; for tens to hundreds of accounts, TGW can be more cost‑effective and operationally simpler despite the per‑attachment and per‑GB fees.

11.4 CloudWatch log and monitoring costs

CloudWatch Logs: DMS streams logs to CloudWatch, which you can retain for a configured period.
CloudWatch Alarms and Metrics: alarms are cheap per metric, but if you create hundreds of DMS‑task‑specific alarms, costs grow.
EventBridge: rules are low‑cost per‑million events, but complex routing can increase complexity.

Best practice: use log‑retention policies (for example, 30 days) and reuse alarm templates for CDC‑latency and CPU across tasks.

11.5 Cross‑AZ transfer and CDC runtime impact

Data transferred across Availability Zones (for example, DMS instance in us‑east‑1a and DB in us‑east‑1b) is usually free within the same region, but it still adds latency.
For long‑running CDC workloads, CDC latency is driven by:
- Quantity and size of transactions.
- Schema design (large tables, LOBs, triggers, etc.).
- DMS instance size and network path.
Larger DMS instances reduce latency but increase hourly cost; for some workloads, it may be cheaper to run CDC for a longer period on a smaller instance than upgrade to a large instance for a short burst.

12. Common DMS Limits / Pitfalls

AWS DMS enforces quotas per account and region; exceeding them causes API errors or resource creation failures.

12.1 Key limits

Tasks per replication instance: up to 200 tasks.
Tasks per account: up to 600 tasks.
Replication instances per account: up to 60 instances.
Endpoints per account: up to 1,000 endpoints.
API request throttling: 200 requests per second, with 8‑request‑per‑second refresh rate.

If you run many parallel migrations, plan to distribute tasks across multiple DMS instances or accounts.

12.2 CDC and large‑table behavior

CDC latency can spike if the DMS instance cannot keep up with the transaction rate.
Large tables can slow down full‑load; DMS may split the table into multiple sessions, but that still consumes memory and network.
LOB handling (for example, very large CLOBs or BLOBs) can exhaust DMS storage or cause CDC delays.

12.3 Unsupported features and engines

DMS does not support all engines and all features:

Some SQL Server row‑size changes above 8000 bytes can be mishandled.
Some advanced features such as cross‑engine triggers, complex stored‑procedure logic, and certain column‑types may not migrate correctly.

Always run a test‑migration and inspect CDC behavior before cutover.

13. Conclusion

Cross‑account DMS deployments are a straightforward conceptually but require careful attention to networking, IAM, and cost.

VPC peering is the simplest option for one‑to‑one account links.
Transit Gateway scales better for multi‑account enterprises but adds cost.
Public‑internet‑based setups work when peering is not allowed, but they demand strict IP‑based security‑group rules and automation.
Terraform with multi‑account providers, remote state, and workspaces can codify and repeat this pattern across environments.
Monitoring, alarms, and EventBridge notifications protect production CDC workloads from latency spikes or failures.
Cost‑aware engineering—right‑sizing DMS instances, minimizing NAT‑gateway usage, and respecting DMS limits—ensures that migrations are both fast and affordable.

Implemented DevSecOps Pipeline: Integrating CodePipeline, CodeBuild, Container Scanning & Automated Compliance Validation

Manish Kumar — Tue, 24 Feb 2026 08:07:01 +0000

The Challenge

I was approached by a mid-sized fintech company — let's call them PayStream Solutions — that was processing roughly 4 million transactions daily across a microservices architecture running on ECS Fargate. On the surface, they had a functioning CI/CD pipeline. Engineers could push code and see it running in production within about 45 minutes. That sounds fine until you look at what was actually happening inside that pipeline.

There was no security. Not "minimal security" — I mean literally no security gates. Developers pushed Docker images directly to ECR without any vulnerability scanning. Infrastructure changes were applied manually by a senior engineer who had AdministratorAccess on the production account. Secrets were hardcoded in environment variables visible in the CodeBuild console logs. And the compliance team was running quarterly manual audits that produced 80-page PDF reports nobody read.

The specific pain points the CTO articulated when I sat down with the leadership team:

Regulatory pressure: PayStream was preparing for PCI-DSS Level 1 certification. Their QSA (Qualified Security Assessor) had flagged the lack of continuous compliance monitoring as a critical gap.
Incident aftermath: Three months prior, a developer accidentally pushed a container image built on an ubuntu:20.04 base that carried 14 known HIGH-severity CVEs. Nobody caught it. It ran in production for 11 days before a routine scan surfaced it.
No audit trail: When the compliance team asked "who deployed what, when, and what was the security posture at deploy time?", the answer was a shrug and a CloudTrail log nobody knew how to read.
Slow feedback loops: Security findings from their periodic scans took weeks to route back to developers, by which time the affected code had already been superseded three times.
Budget constraints: They had a hard ceiling of roughly $8,000/month for the entire DevOps toolchain, which ruled out third-party SAST/DAST platforms like Veracode or Checkmarx in their enterprise tiers.

The timeline pressure was acute — the PCI-DSS audit was scheduled in 90 days. That gave me roughly 12 weeks to design, implement, test, and document the entire pipeline transformation. It was tight but doable with the right architecture decisions upfront.

Initial Assessment

I spent the first week doing nothing but observing and asking uncomfortable questions. I've found that the most dangerous assumptions in any infrastructure engagement are the ones everyone considers "obviously true." My job was to challenge those.

What I discovered during the analysis:

I pulled up the existing CodePipeline configuration and found a two-stage setup: Source (CodeCommit) → Deploy (manual ECS deploy via CLI script embedded in a CodeBuild project). That's it. No test stage. No build validation. The "build" step was literally docker build && docker push. The deploy step was a aws ecs update-service call.

Here's what the metrics told me:

Mean time to detect (MTTD) a vulnerability: 47 days (based on the last four incidents)
Mean time to remediate (MTTR): 18 days after detection
Pipeline execution time: 43 minutes average (almost all of it was the ECS service update waiting for health checks)
False production deployments per month: 6 — meaning code that failed basic functional tests still reached production because there were no automated gates
Manual security review overhead: Two engineers spending ~30% of their time on security-adjacent work that should have been automated

I interviewed the DevOps lead, two senior developers, the CISO, and the compliance manager. The CISO's exact words were: "I don't know what's running in my containers right now, and that scares me." That sentence became the north star for the entire engagement.

The compliance manager showed me their current evidence collection process for audits — a spreadsheet with manual screenshots. My reaction was visceral. They needed evidence to be machine-generated, timestamped, immutable, and linkable to specific pipeline executions.

Risk factors I identified:

No separation between the IAM roles used for CI and those used for CD — the build role could deploy to production directly
Secrets Manager was not being used; credentials were in Parameter Store as plaintext strings
ECR repositories had no lifecycle policies, so image bloat was costing approximately $340/month in unnecessary S3 storage
No VPC Flow Logs were enabled on the production VPC — a PCI-DSS requirement
CloudTrail was enabled but logs were flowing to a bucket in the same account, making them susceptible to tampering
No GuardDuty, no Security Hub, no Config Rules — essentially no detective control layer whatsoever

Solution Design

The architecture I proposed centered on a single principle: security as a first-class pipeline citizen, not an afterthought bolted on at the end. Every gate in the pipeline had to be automated, auditable, and fail-closed — meaning if a security check couldn't run, the pipeline failed. Not warned. Failed.

AWS Services Selected (and Why)

AWS CodePipeline (V2) was the natural orchestration backbone. The V2 version introduced variables and triggers that the older V1 lacked, which was important for passing scan results between pipeline stages without writing to S3 and reading back. CodePipeline also integrates natively with EventBridge, which meant I could fire compliance events without custom Lambda glue code.

AWS CodeBuild handled all the compute-intensive security scanning tasks. The key architectural decision here was splitting security scanning into separate CodeBuild projects rather than stuffing everything into one monolithic buildspec. This gave us independent scalability, cleaner failure attribution ("the SAST stage failed, not the build stage"), and cheaper retries on transient failures.

Amazon ECR with Enhanced Scanning (Amazon Inspector v2) replaced the basic CVE scanning that was previously disabled. Enhanced scanning provides continuous monitoring — not just on-push — and sends findings to Security Hub and EventBridge automatically. The basic scanning uses the Clair project database, while enhanced scanning uses Inspector's more comprehensive intelligence that includes OS packages and programming language packages.

AWS Security Hub became the single pane of glass for all security findings. Inspector, GuardDuty, Config, and our custom CodeBuild scan results all funneled into Security Hub using the AWS Security Finding Format (ASFF). This was critical for the PCI-DSS audit — one place to pull evidence from.

AWS Config + Conformance Packs handled the automated compliance validation layer. Config continuously evaluates resource configurations against rules. CodePipeline can query Config compliance status as a pipeline gate — if your infrastructure drift check fails, the deployment doesn't proceed.

AWS Secrets Manager replaced every hardcoded credential and Parameter Store plaintext value. Secrets Manager supports automatic rotation, which is a hard requirement for PCI-DSS.

AWS KMS (Customer Managed Keys) was used to encrypt everything: CodePipeline artifacts in S3, CodeBuild environment variables, ECR images, and CloudWatch Logs. Using CMKs rather than AWS-managed keys gave us fine-grained control over who could decrypt what and when — critical for the separation-of-duties requirement.

Amazon Inspector v2 provided the InspectorScan action that is now natively available in CodePipeline, which can run both source code scans and ECR image scans as first-class pipeline actions without any custom CodeBuild glue.

AWS Systems Manager Session Manager replaced all SSH/bastion access. No port 22, no key pairs, no bastion hosts — Session Manager provides browser-based or CLI shell access to EC2 and ECS container instances with full session logging to CloudWatch and S3.

Architecture Description

Cost vs. Performance Trade-offs

The most contentious design discussion was around CodeBuild compute sizing. Running security scans on BUILD_GENERAL1_LARGE instances (4 vCPU, 7 GB) versus BUILD_GENERAL1_MEDIUM (2 vCPU, 3.75 GB) was roughly a 2x cost difference per build minute. I ran the math with the team: at roughly 80 pipeline executions per day, the SAST scan stage alone would cost ~$180/month on LARGE versus ~$90/month on MEDIUM. We went with MEDIUM for all scan stages and LARGE only for the Docker build itself, which is the most compute-intensive step. That's a practical FinOps decision that most teams overlook.

For the build environment, I used CodeBuild spot capacity where possible — particularly for the non-blocking SAST stages that could tolerate interruption and retry. AWS Spot capacity for CodeBuild isn't the same as EC2 Spot, but CodeBuild does support fleet-mode with spot capacity that can reduce costs by up to 70%.

Security and Compliance Requirements Addressed

PayStream needed to satisfy PCI-DSS requirements 6.3 (vulnerability scanning), 6.4 (security gates in SDLC), 8.2 (credential management), 10.2 (audit logging), and 11.3 (penetration testing integration). The architecture addressed each of these through automated pipeline stages rather than manual controls.

In-Depth Discussion of Key Areas

FinOps: Embedding Cost Governance Into the Pipeline

One thing I've learned across several engagements is that FinOps and DevSecOps intersect more than people realize. The pipeline itself is a cost center — every build minute, every artifact stored, every Lambda invocation for a compliance check costs money.

Here's how we embedded cost awareness:

Tagging enforcement via Config Rules: Every resource deployed through the pipeline had to carry Environment, Project, CostCenter, and Owner tags. A Config managed rule (required-tags) evaluated this on every resource change. Non-compliant resources triggered a Security Hub finding and an EventBridge notification to the cost owner.
ECR Lifecycle Policies: This was an immediate quick win. I implemented lifecycle policies to expire untagged images after 1 day and keep only the last 10 tagged images per repository. This alone reduced ECR storage costs from $340/month to under $40/month.

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Expire untagged images after 1 day",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 1
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Keep last 10 tagged images",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["v"],
        "countType": "imageCountMoreThan",
        "countNumber": 10
      },
      "action": { "type": "expire" }
    }
  ]
}

CodeBuild artifact caching: The dependency download phase was responsible for ~40% of build time. Enabling S3-backed cache for pip install and npm install reduced average build time from 43 minutes to 17 minutes — which in turn cut CodeBuild costs almost proportionally.
AWS Cost Anomaly Detection: I set up alerts for >20% week-over-week cost increases in the CodePipeline cost category. This catches situations like a runaway retry loop where a misconfigured pipeline triggers thousands of builds per hour.
AWS Compute Optimizer recommendations: After two weeks of pipeline operation, Compute Optimizer data showed that our CodeBuild LARGE instances were consistently using only 45% of CPU during Docker builds. I right-sized three stages down to MEDIUM, saving another $60/month.

💡 Pro Tip: Use AWS Cost Allocation Tags from day one. Retroactively tagging resources is painful and usually incomplete. In a DevSecOps context, tags serve double duty — they're both FinOps tools and security evidence artifacts.

AWS Well-Architected Framework (WAF) Application

The Six Pillars of the Well-Architected Framework aren't just a compliance checkbox — they're a design forcing function. Here's how each pillar manifested in this project:

Operational Excellence: All pipeline stages produced structured JSON output written to S3. I configured CloudWatch Dashboards with pipeline health metrics — MTTR, deployment frequency, change failure rate, and lead time. These are the four DORA metrics, and having them automated meant the CISO could see security health at a glance without asking engineers.

Security (the primary pillar for this engagement): Every action in the pipeline ran under a purpose-specific IAM role with the minimum permissions needed. The CodeBuild role for SAST could only read from the source bucket and write to the artifacts bucket. It had zero IAM, EC2, or ECS permissions. The deploy role could update ECS services but couldn't modify IAM policies. This is the least-privilege principle applied concretely.

Reliability: Pipeline stages had retry logic configured (CodePipeline supports up to 5 retries per action). The Config conformance pack deployment used an idempotent CloudFormation template, so re-running it produced the same result regardless of current state.

Performance Efficiency: By parallelizing the SAST scan and the Dockerfile linting into concurrent CodeBuild actions within the same pipeline stage, I cut the security scanning wall-clock time from sequential 18 minutes to parallel 11 minutes.

Cost Optimization: Covered in the FinOps section above.

Sustainability: Fewer build minutes through caching and right-sizing = less compute energy consumed. It's a secondary benefit, but AWS Sustainability pillar guidance specifically calls out compute right-sizing as a key lever.

AWS Security Reference Architecture (SRA) Application

The AWS SRA provides prescriptive guidance for deploying security services in a multi-account AWS Organizations environment. PayStream was a single-account shop when I found them. My first recommendation — and it was non-negotiable from a PCI-DSS standpoint — was to migrate to a multi-account structure.

The SRA-aligned account structure I implemented:

Account	Purpose	Key Services
Management (Root)	SCPs, billing, AWS Organizations	AWS Organizations, Service Control Policies
Security Tooling	Centralized security services	Security Hub (delegated admin), GuardDuty, Config aggregator, CloudTrail Lake
Log Archive	Immutable centralized logging	S3 (Object Lock, MFA delete), CloudWatch Logs
Shared Services	Shared pipeline infrastructure	CodePipeline, CodeBuild, ECR, Secrets Manager
Dev Workload	Development environment	ECS, RDS, VPC
Staging Workload	Pre-production environment	ECS, RDS, VPC
Prod Workload	Production environment	ECS, RDS, VPC (strict SCPs)

The critical SRA concept I explained to the team is delegated administration. Rather than enabling Security Hub in every account individually, you designate the Security Tooling account as the Security Hub administrator. All member accounts automatically send findings there. This means even if a developer accidentally disables Security Hub in their account (which I've seen happen), the Security Tooling account still has a complete record of all prior findings.

Service Control Policies at the root OU level enforced guardrails that no individual account could override:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyLeavingOrganization",
      "Effect": "Deny",
      "Action": "organizations:LeaveOrganization",
      "Resource": "*"
    },
    {
      "Sid": "DenyDisableSecurityServices",
      "Effect": "Deny",
      "Action": [
        "guardduty:DeleteDetector",
        "guardduty:DisassociateFromMasterAccount",
        "securityhub:DisableSecurityHub",
        "config:DeleteConfigRule",
        "config:StopConfigurationRecorder",
        "cloudtrail:DeleteTrail",
        "cloudtrail:StopLogging"
      ],
      "Resource": "*"
    },
    {
      "Sid": "RequireIMDSv2",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringNotEquals": {
          "ec2:MetadataHttpTokens": "required"
        }
      }
    }
  ]
}

AWS Systems Manager Session Manager

Eliminating SSH was one of the first things I did — and the one developers pushed back on most. The objection is always "but how do I debug a running container?"

Session Manager answers that question completely. Instead of opening port 22 on a security group, you install the SSM Agent on your EC2 instances (it comes pre-installed on Amazon Linux 2 and Amazon Linux 2023) and give the instance profile the AmazonSSMManagedInstanceCore managed policy. That's it. No inbound rules needed in the security group at all.

For ECS containers, you enable ECS Exec, which tunnels through Session Manager to give you a shell inside a running container:

# Enable ECS Exec on a service
aws ecs update-service \
  --cluster paystream-prod \
  --service payment-api \
  --enable-execute-command

# Connect to a running container
aws ecs execute-command \
  --cluster paystream-prod \
  --task <task-id> \
  --container payment-api \
  --interactive \
  --command "/bin/bash"

Every Session Manager session is automatically logged to CloudWatch Logs and S3. The logs capture the full command history with timestamps and the IAM principal who initiated the session. For PCI-DSS audit purposes, this is gold — you have a complete, tamper-evident record of every interactive access to production systems.

⚠️ Gotcha: Session Manager requires the instance to have outbound HTTPS (port 443) access to the SSM endpoints — either via an internet gateway, NAT gateway, or VPC Interface Endpoints for SSM. In a private VPC with no internet access, you need three VPC endpoints: com.amazonaws.region.ssm, com.amazonaws.region.ssmmessages, and com.amazonaws.region.ec2messages. I learned this the hard way on the staging environment when Session Manager silently failed to connect and I spent two hours assuming it was an IAM issue.

VPC Flow Logs: Network Visibility and Compliance

VPC Flow Logs capture metadata about accepted and rejected IP traffic flowing through your VPC's ENIs (Elastic Network Interfaces). They don't capture packet payloads — just the "who talked to whom, on what port, was it accepted or rejected, and how many bytes" metadata. That metadata is surprisingly powerful for compliance.

For PCI-DSS, requirement 10.2 mandates logging of all access to cardholder data, which means network traffic to and from the payment processing services needed to be logged. Flow Logs satisfied this requirement automatically.

Here's the Terraform to enable VPC Flow Logs at the VPC level, publishing to both CloudWatch Logs and S3 (dual-destination for redundancy and different retention policies):

resource "aws_flow_log" "paystream_vpc_flow_log" {
  vpc_id          = aws_vpc.paystream_prod.id
  traffic_type    = "ALL"  # Capture ACCEPT, REJECT, and ALL traffic
  iam_role_arn    = aws_iam_role.flow_log_role.arn
  log_destination = aws_cloudwatch_log_group.vpc_flow_logs.arn
  log_format      = "${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${windowstart} ${windowend} ${action} ${tcp-flags} $${flow-direction}"
}

resource "aws_flow_log" "paystream_vpc_flow_log_s3" {
  vpc_id               = aws_vpc.paystream_prod.id
  traffic_type         = "ALL"
  log_destination_type = "s3"
  log_destination      = "${aws_s3_bucket.flow_logs_archive.arn}/vpc-flow-logs/"
  log_format           = "${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${windowstart} ${windowend} ${action} ${tcp-flags} $${flow-direction}"
}

resource "aws_cloudwatch_log_group" "vpc_flow_logs" {
  name              = "/aws/vpc/flowlogs/paystream-prod"
  retention_in_days = 90  # PCI-DSS requires 1 year; use S3 for longer-term archival
  kms_key_id        = aws_kms_key.cloudwatch_key.arn
}

I also set up a CloudWatch Metric Filter and Alarm to detect port scanning patterns — a sudden spike in REJECT records from a single source IP within a 5-minute window triggers a GuardDuty finding and an SNS alert to the security team.

One thing that often surprises people: Flow Logs have a small but measurable cost. At PayStream's traffic volume (~2 GB of flow log data per day), the CloudWatch Logs ingestion cost was approximately $1/GB = ~$60/month. I moved logs older than 14 days to S3 Glacier Instant Retrieval, which reduced the total storage cost to under $15/month while maintaining the 90-day hot access period required for active investigation.

KMS Key Policies: Encryption Architecture

AWS KMS with Customer Managed Keys (CMKs) is one of those areas where the gap between "it works" and "it's properly secured" is enormous, and most teams live in "it works" territory.

The key insight about KMS key policies is that they're resource-based policies — they operate at the key level regardless of what IAM policies say. If a KMS key policy says "deny everyone except the key admin role from using this key," then even a user with AdministratorAccess on the account cannot use that key. This is fundamentally different from most AWS resources where IAM identity policies can override resource-based policies.

For PayStream, I created purpose-specific CMKs with tight key policies:

{
  "Version": "2012-10-17",
  "Id": "paystream-codepipeline-artifacts-key-policy",
  "Statement": [
    {
      "Sid": "EnableIAMUserPermissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_ID:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "AllowCodePipelineServiceRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_ID:role/CodePipelineServiceRole"
      },
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowCodeBuildServiceRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_ID:role/CodeBuildSecurityScanRole"
      },
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyUnencryptedObjectUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "kms:Decrypt",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "kms:CallerAccount": "ACCOUNT_ID"
        }
      }
    }
  ]
}

An important distinction people often miss: kms:* in the key policy only works if an IAM identity policy also grants the permission. For IAM users and roles (non-root principals), both the KMS key policy AND the IAM identity policy must allow the action. For the account root principal, the key policy alone is sufficient. This is a frequent source of confusion.

For cross-account scenarios (the deploy role in the Prod account needs to decrypt artifacts encrypted by the Shared Services account KMS key), you need to grant permissions in two places: the KMS key policy must list the external role's ARN, and the role's IAM policy in the target account must grant the KMS decrypt action.

💡 Pro Tip: Enable KMS key rotation (annual automatic rotation) and set up CloudWatch Alarms on kms:Decrypt calls that significantly exceed the baseline. A sudden spike in decrypt calls can indicate credential compromise where an attacker is exfiltrating data.

IAM Permission Boundary Mechanics

Permission boundaries are one of the most misunderstood IAM concepts, but in a DevSecOps context they're indispensable — particularly for CI/CD pipelines that need to create IAM roles.

The core problem they solve: Your CodePipeline deploy stage needs to create IAM roles for new Lambda functions or ECS task definitions. But if the deploy role has iam:CreateRole, it can theoretically create a role with AdministratorAccess — which is a privilege escalation path. This is why security teams often block CI/CD pipelines from touching IAM at all, which breaks modern IaC workflows.

The permission boundary solution: A permission boundary is a managed IAM policy that you attach to a role, which acts as a ceiling on what permissions that role can ever have — regardless of what policies are directly attached to it. The effective permissions are the intersection of what the identity policy allows AND what the boundary allows.

Here's how I implemented this for PayStream's Terraform deploy role:

# The permission boundary policy - defines the MAXIMUM permissions
# any role created by the deploy pipeline can have
resource "aws_iam_policy" "devsecops_permission_boundary" {
  name        = "DevSecOpsDeploymentBoundary"
  description = "Maximum permissions for roles created via CI/CD pipeline"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowECSAndLambdaPermissions"
        Effect = "Allow"
        Action = [
          "ecs:*",
          "lambda:*",
          "s3:GetObject",
          "s3:PutObject",
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "xray:PutTraceSegments"
        ]
        Resource = "*"
      },
      {
        Sid    = "DenyPrivilegeEscalation"
        Effect = "Deny"
        Action = [
          "iam:CreateUser",
          "iam:AttachUserPolicy",
          "iam:PutUserPolicy",
          "organizations:*",
          "account:*"
        ]
        Resource = "*"
      }
    ]
  })
}

# The deploy role itself - must pass the boundary when creating child roles
resource "aws_iam_role" "codepipeline_deploy_role" {
  name                 = "CodePipelineDeployRole"
  permissions_boundary = aws_iam_policy.devsecops_permission_boundary.arn

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Service = "codepipeline.amazonaws.com"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

The critical mechanic: when this deploy role creates a new IAM role via Terraform (aws_iam_role), the new role automatically inherits the permission boundary. The deploy role's IAM policy includes iam:CreateRole but only with a condition that enforces the boundary:

{
  "Sid": "AllowCreateRoleWithBoundary",
  "Effect": "Allow",
  "Action": "iam:CreateRole",
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "iam:PermissionsBoundary": "arn:aws:iam::ACCOUNT_ID:policy/DevSecOpsDeploymentBoundary"
    }
  }
}

Without this condition, the deploy role could create a role without a boundary. The condition locks it down so every role created through the pipeline must wear the same constrictive boundary. It's an elegant mechanism.

⚠️ Gotcha: Permission boundaries do NOT grant permissions — they only constrain them. A common mistake is treating a permission boundary as an "allow list" and wondering why the role still can't do things that are in the boundary policy. The boundary is a ceiling; the floor is set by the identity policy attached to the role.

Disaster Recovery with RTO/RPO

While this engagement was primarily about DevSecOps, a complete production architecture had to address DR. PayStream's previous DR plan was "we have daily RDS snapshots, good luck." That's not a plan. I established formal RTO and RPO targets in collaboration with the business team:

Recovery Time Objective (RTO): 4 hours (how long the business can tolerate being down)
Recovery Point Objective (RPO): 15 minutes (how much data they can afford to lose)

The 15-minute RPO drove several architectural decisions:

For RDS (Aurora PostgreSQL): I enabled Aurora Global Database with a secondary region (ap-south-1, Mumbai, given PayStream's India-focused user base). Aurora replication typically achieves sub-second replication lag, well within the 15-minute RPO. Global Database failover (managed failover) completes in roughly 1 minute, which also easily satisfies the 4-hour RTO.

For the pipeline artifacts and state: S3 buckets holding pipeline artifacts were configured with Cross-Region Replication (CRR) to the DR region. ECR images were pushed to both regions simultaneously using a post-build CodeBuild step. This ensured the DR region could deploy the latest image without pulling from the primary region.

For the pipeline itself: The CodePipeline and CodeBuild configuration was maintained as Terraform code in a Git repository. In a disaster scenario, re-creating the pipeline in the DR region was a terraform apply away — a process that took approximately 8 minutes in our DR runbook test.

DR testing was integrated into the pipeline itself. Quarterly, a RunDrTest CodePipeline execution would:

Spin up a test ECS cluster in the DR region using the latest DR image
Run a synthetic transaction suite against it
Verify database connectivity to the Aurora Global secondary
Publish a DR test report to S3 and a Security Hub finding with the results
Tear down the test infrastructure

This meant the DR plan was validated continuously, not just during the annual DR exercise that most companies do (and usually fake).

Implementation Journey

Phase 1: Foundation

The first two weeks were entirely about infrastructure foundations. No application code, no pipeline stages — just the bedrock that everything else would sit on.

VPC Design:
I designed a three-tier VPC architecture with strict subnet separation:

Production VPC: 10.0.0.0/16

Public Subnets (10.0.1.0/24, 10.0.2.0/24, 10.0.3.0/24)
  → ALB, NAT Gateways only. No application instances here.

Private App Subnets (10.0.10.0/24, 10.0.11.0/24, 10.0.12.0/24)
  → ECS Fargate tasks (payment API, transaction processor)
  → Route table: 0.0.0.0/0 → NAT Gateway

Private Data Subnets (10.0.20.0/24, 10.0.21.0/24, 10.0.22.0/24)
  → Aurora PostgreSQL cluster
  → ElastiCache (Redis)
  → No internet route whatsoever

Pipeline Subnet (10.0.30.0/24)
  → CodeBuild VPC network interface (for scans needing VPC access)
  → VPC Interface Endpoints for AWS services

VPC Flow Logs were enabled before any application traffic flowed through the VPC. I've made the mistake before of enabling Flow Logs after the fact and losing the forensic baseline for "normal" traffic patterns.

IAM Baseline:
I created a purpose-specific IAM role hierarchy before writing a single line of application code:

CodePipelineOrchestrationRole — can read/write artifacts S3, start CodeBuild, pass roles to CodeBuild/ECS
CodeBuildSASTRole — read-only S3 artifacts, no AWS API calls except CloudWatch Logs write
CodeBuildBuildRole — ECR push, S3 artifacts, CloudWatch Logs
CodeBuildComplianceRole — Config read, Security Hub write, CloudWatch Logs
ECSTaskExecutionRole — ECR pull, Secrets Manager read, CloudWatch Logs
ECSTaskRole — application-specific permissions only (DynamoDB, SQS, etc.)

The initial challenge was explaining to the team why there were 6 roles for a single pipeline. The answer is that blast radius control is worth the setup overhead. If CodeBuildSASTRole is compromised (e.g., a malicious dependency in a SAST tool), the attacker can read build artifacts but cannot push images, touch ECS, or read secrets.

Phase 2: Core Services

With the foundation in place, I built the actual pipeline stages.

Stage 1: Source

I connected GitHub (migrating away from CodeCommit given GitHub's richer webhook support) using a CodeStar Connection. This kept credentials out of CodeBuild entirely — the connection is a service-managed credential stored securely by AWS.

Branch protection rules on GitHub enforced:

All PRs require at least one approved review
Status checks must pass (the SAST pipeline ran on every PR, not just merges to main)
No direct pushes to main or release/* branches

Stage 2: Static Analysis Security Testing (SAST)

This stage ran three concurrent CodeBuild actions:

# buildspec-sast.yml
version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - pip install bandit semgrep checkov
      - npm install -g @hadolint/hadolint
  build:
    commands:
      # Python SAST with Bandit
      - bandit -r ./src -f json -o bandit-report.json || true

      # Semgrep with OWASP ruleset
      - semgrep scan --config=p/owasp-top-ten 
          --config=p/python 
          --json 
          --output=semgrep-report.json 
          ./src || true

      # Dockerfile linting
      - hadolint Dockerfile --format json > hadolint-report.json || true

      # IaC scanning for Terraform
      - checkov -d ./terraform 
          --framework terraform 
          -o json 
          --output-file checkov-report.json || true
  post_build:
    commands:
      # Fail on HIGH/CRITICAL findings
      - python evaluate_sast_results.py
      # Upload reports to S3 audit bucket
      - aws s3 cp bandit-report.json s3://paystream-audit-reports/sast/$CODEBUILD_BUILD_ID/
      - aws s3 cp semgrep-report.json s3://paystream-audit-reports/sast/$CODEBUILD_BUILD_ID/
artifacts:
  files:
    - '**/*-report.json'
  base-directory: '.'

The evaluate_sast_results.py script parsed each report, applied a configurable severity threshold (CRITICAL always fails, HIGH fails unless explicitly suppressed with a JIRA ticket reference in a .security-exceptions.yaml file), and exited with code 1 if the threshold was exceeded.

Stage 3: Container Build and Scanning

This was the heart of the pipeline:

# buildspec-build-scan.yml
version: 0.2
env:
  secrets-manager:
    SONAR_TOKEN: "arn:aws:secretsmanager:ap-south-1:ACCOUNT:secret:sonar-token"
phases:
  pre_build:
    commands:
      - REPOSITORY_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/paystream-payment-api
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=v${CODEBUILD_BUILD_NUMBER}-${COMMIT_HASH}
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | 
          docker login --username AWS --password-stdin $REPOSITORY_URI
  build:
    commands:
      # Multi-stage build with explicit base image pinning
      - docker build 
          --build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
          --build-arg VCS_REF=$COMMIT_HASH
          --label "build.id=$CODEBUILD_BUILD_ID"
          -t $REPOSITORY_URI:$IMAGE_TAG 
          -f Dockerfile .
      - docker push $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      # Wait for ECR Enhanced Scanning (Inspector v2) to complete
      - |
        echo "Waiting for Inspector scan..."
        for i in $(seq 1 30); do
          STATUS=$(aws ecr describe-image-scan-findings \
            --repository-name paystream-payment-api \
            --image-id imageTag=$IMAGE_TAG \
            --query 'imageScanStatus.status' \
            --output text 2>/dev/null || echo "IN_PROGRESS")
          if [ "$STATUS" = "COMPLETE" ]; then
            echo "Scan complete after ${i} checks"
            break
          fi
          echo "Scan in progress... ($i/30)"
          sleep 20
        done
      # Parse and gate on findings
      - |
        CRITICAL=$(aws ecr describe-image-scan-findings \
          --repository-name paystream-payment-api \
          --image-id imageTag=$IMAGE_TAG \
          --query 'imageScanFindings.findingCounts.CRITICAL' \
          --output text)
        HIGH=$(aws ecr describe-image-scan-findings \
          --repository-name paystream-payment-api \
          --image-id imageTag=$IMAGE_TAG \
          --query 'imageScanFindings.findingCounts.HIGH' \
          --output text)
        echo "Critical: $CRITICAL, High: $HIGH"
        if [ "$CRITICAL" != "None" ] && [ "$CRITICAL" -gt 0 ]; then
          echo "PIPELINE BLOCKED: Critical vulnerabilities found"
          exit 1
        fi
        if [ "$HIGH" != "None" ] && [ "$HIGH" -gt 5 ]; then
          echo "PIPELINE BLOCKED: Too many HIGH vulnerabilities ($HIGH)"
          exit 1
        fi
      # Write image metadata for downstream stages
      - printf '[{"name":"payment-api","imageUri":"%s"}]' 
          $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
artifacts:
  files:
    - imagedefinitions.json

Stage 4: Automated Compliance Validation

This stage queried AWS Config to verify that the target environment (staging or production) was in a compliant state before deploying to it:

# compliance_gate.py
import boto3
import sys

def check_compliance_before_deploy(environment):
    config = boto3.client('config')

    # Check the Operational Best Practices for PCI DSS conformance pack
    response = config.describe_compliance_by_config_rule(
        ComplianceTypes=['NON_COMPLIANT']
    )

    non_compliant_critical = [
        r for r in response['ComplianceByConfigRules']
        if r['ConfigRuleName'].startswith('paystream-critical-')
        and r['Compliance']['ComplianceType'] == 'NON_COMPLIANT'
    ]

    if non_compliant_critical:
        print(f"❌ COMPLIANCE GATE FAILED: {len(non_compliant_critical)} critical rules non-compliant")
        for rule in non_compliant_critical:
            print(f"  - {rule['ConfigRuleName']}: {rule['Compliance']['ComplianceType']}")
        sys.exit(1)

    print(f"✅ Compliance gate passed for {environment}")
    return True

if __name__ == '__main__':
    check_compliance_before_deploy(sys.argv if len(sys.argv) > 1 else 'staging')

The conformance pack I deployed included both AWS Managed Config Rules and custom rules specific to PayStream's PCI-DSS requirements:

# paystream-pci-conformance-pack.yaml
Parameters:
  AccessKeysRotatedParamMaxAccessKeyAge:
    Default: '90'
    Type: String
Resources:
  # PCI Req 8.3 - MFA for all non-console access
  MFAEnabledForIAMConsolAccess:
    Properties:
      ConfigRuleName: paystream-critical-mfa-console-access
      Source:
        Owner: AWS
        SourceIdentifier: MFA_ENABLED_FOR_IAM_CONSOLE_ACCESS
    Type: AWS::Config::ConfigRule

  # PCI Req 6.3 - Vulnerability management
  ECRImageScanningEnabled:
    Properties:
      ConfigRuleName: paystream-critical-ecr-scan-on-push
      Source:
        Owner: AWS
        SourceIdentifier: ECR_PRIVATE_IMAGE_SCANNING_ENABLED
    Type: AWS::Config::ConfigRule

  # PCI Req 10.3 - VPC Flow Logs
  VPCFlowLogsEnabled:
    Properties:
      ConfigRuleName: paystream-critical-vpc-flow-logs
      Source:
        Owner: AWS
        SourceIdentifier: VPC_FLOW_LOGS_ENABLED
    Type: AWS::Config::ConfigRule

  # PCI Req 3.4 - Encryption at rest
  S3BucketServerSideEncryptionEnabled:
    Properties:
      ConfigRuleName: paystream-critical-s3-encryption
      Source:
        Owner: AWS
        SourceIdentifier: S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED
    Type: AWS::Config::ConfigRule

  # Secrets Manager rotation enabled
  SecretsManagerRotationEnabled:
    Properties:
      ConfigRuleName: paystream-critical-secrets-rotation
      Source:
        Owner: AWS
        SourceIdentifier: SECRETSMANAGER_ROTATION_ENABLED_CHECK
    Type: AWS::Config::ConfigRule

Phase 3: Advanced Features

Monitoring and Observability Stack:

I deployed a three-layer observability stack:

Infrastructure metrics: CloudWatch Container Insights for ECS (CPU, memory, network, disk per task), custom CloudWatch Metrics from application code published via the EMF (Embedded Metric Format) library
Application logs: Structured JSON logs via FireLens (AWS's log routing solution built on Fluent Bit) routing to CloudWatch Logs and OpenSearch Service
Distributed tracing: AWS X-Ray with sampling rules set to 5% for normal traffic and 100% for requests flagged with specific transaction IDs

The Security Dashboard in CloudWatch combined pipeline execution success/failure rates, Security Hub finding trends, Config compliance scores, and MTTD/MTTR metrics into a single operations view.

Auto-Scaling Configuration:

ECS Service Auto Scaling was configured with a multi-metric policy:

Target tracking on CPU utilization (target: 60%)
Step scaling on custom metric: SQS queue depth per ECS task (scale out when >500 messages/task)
Scheduled scaling for known peak periods (payment processing business hours)

Minimum task count was set to 3 (spanning all three AZs) to ensure High Availability even before auto-scaling kicks in.

Disaster Recovery Implementation:

As described in the DR section above, I implemented Aurora Global Database with automated failover testing integrated into the pipeline's quarterly schedule.

Phase 4: Optimization

Performance Tuning:

After two weeks in production, I analyzed X-Ray traces and found that 35% of payment API latency was database connection establishment overhead. The solution was Amazon RDS Proxy, which maintains a persistent connection pool and reduces connection time from ~150ms to ~2ms for typical ECS tasks that start and stop frequently.

X-Ray trace analysis also revealed that the container image pull time on cold ECS task starts was 45 seconds for a 1.2 GB image. I addressed this through two measures:

Multi-stage Docker builds that reduced the final image size from 1.2 GB to 280 MB
ECR pull-through cache configuration to ensure images were always warm in the local ECR endpoint

Cost Optimization Measures:

Moved non-production environments to ECS Fargate Spot — 70% compute cost reduction for dev and staging
Implemented CloudWatch Logs tiering: 14 days in CloudWatch (hot), then S3 Standard-IA (warm), then Glacier Instant Retrieval after 90 days (cold)
Used AWS Compute Optimizer recommendations to right-size ECS task CPU/memory allocations — found several tasks over-provisioned by 40%
Enabled S3 Intelligent-Tiering on the audit reports bucket, which automatically moved infrequently accessed compliance reports to cheaper tiers

Security Hardening:

Enabled Amazon Macie on all S3 buckets storing PII data (payment records) — Macie automatically discovers and alerts on sensitive data
Configured GuardDuty with Malware Protection for ECS, which scans ECS task filesystem volumes on suspicious activity triggers
Implemented AWS WAF in front of the ALB with the AWS Managed Rules group for common vulnerabilities (SQLi, XSS, bot control)
Set ECR repository policies to deny image pull from outside the account unless explicitly authorized

Technical Specifications

Compute

ECS Fargate (Production): Tasks sized at 1 vCPU / 2 GB (payment API), 0.5 vCPU / 1 GB (transaction processor)
ECS Fargate Spot (Staging/Dev): Same sizes, 70% cost reduction
CodeBuild: BUILD_GENERAL1_MEDIUM (2 vCPU, 3.75 GB) for security scans; BUILD_GENERAL1_LARGE (4 vCPU, 7 GB) for Docker build stage
RDS Proxy: Enabled between ECS tasks and Aurora, connection pool max 100 connections

Database

resource "aws_rds_cluster" "paystream_aurora" {
  cluster_identifier     = "paystream-prod"
  engine                 = "aurora-postgresql"
  engine_version         = "15.4"
  database_name          = "paystream"
  master_username        = "paystream_admin"
  manage_master_user_password = true  # Secrets Manager managed password

  vpc_security_group_ids = [aws_security_group.aurora_sg.id]
  db_subnet_group_name   = aws_db_subnet_group.data_tier.name

  storage_encrypted      = true
  kms_key_id             = aws_kms_key.aurora_key.arn

  backup_retention_period    = 35  # PCI-DSS requires 1 year; use exports for long-term
  preferred_backup_window    = "02:00-03:00"
  preferred_maintenance_window = "sun:04:00-sun:05:00"

  enabled_cloudwatch_logs_exports = ["postgresql"]

  deletion_protection = true  # Cannot accidentally delete production DB

  tags = {
    Environment = "production"
    CostCenter  = "platform-engineering"
    DataClass   = "PCI-DSS-Cardholder"
  }
}

Network Architecture

VPC CIDR: 10.0.0.0/16 across 3 AZs (ap-south-1a, ap-south-1b, ap-south-1c)
Security Groups: Allow-listed by service-to-service communication; no broad CIDR-based rules
NACLs: Stateless subnet-level controls supplementing security groups
VPC Endpoints: Interface endpoints for ECR (API + Docker), S3 (Gateway), Secrets Manager, SSM, STS, CloudWatch Logs, KMS — eliminating internet traffic for AWS API calls
Transit Gateway: Connecting workload VPCs to shared services VPC

Terraform Infrastructure Summary

# Key module structure
modules/
  ├── vpc/              # VPC, subnets, route tables, Flow Logs, VPC endpoints
  ├── iam/              # Roles, permission boundaries, policies
  ├── ecr/              # Repositories, lifecycle policies, scanning config
  ├── codepipeline/     # Pipeline stages, actions, artifacts bucket (KMS)
  ├── codebuild/        # Build projects, environments, VPC config
  ├── ecs/              # Cluster, services, task definitions, auto-scaling
  ├── rds/              # Aurora cluster, proxy, subnet group, parameter groups
  ├── security/         # Config rules, conformance pack, Security Hub, GuardDuty
  ├── monitoring/       # CloudWatch dashboards, alarms, metric filters
  └── kms/              # CMKs for each service tier

Challenges and Solutions

Challenge 1: Container Scan False Positives Blocking Deployments

About two weeks after go-live, developers started complaining that pipeline deployments were failing on container scan results for CVEs that had no available fix. The python:3.11-slim base image carried several HIGH-severity CVEs in system libraries where the upstream maintainers had not yet released patches.

How I troubleshot it: I pulled the ECR scan findings for the last 50 failed builds and ran a frequency analysis. 73% of build failures were caused by exactly three CVEs — CVE-2023-XXXX in libssl1.1, CVE-2024-YYYY in zlib, and CVE-2024-ZZZZ in glibc — all with no available fix in the slim variant.

Solution: I implemented a structured exception process. If a CVE has no available fix (verified via the National Vulnerability Database API), developers could submit a suppression entry in a .security-exceptions.yaml file:

suppressions:
  - cve_id: "CVE-2023-XXXX"
    reason: "No available fix in base image; tracked in JIRA-4521"
    expires: "2025-03-01"
    approved_by: "security-team"
    risk_accepted: true

The pipeline script checked expirations and required a JIRA ticket reference. Any suppression older than 90 days automatically became a blocking finding again. This gave developers a legitimate escape valve while maintaining security accountability.

Lesson learned: Zero-tolerance CVE policies sound good but fail in practice. Real security posture comes from risk-based decision making with documented accountability, not blanket blocks.

Challenge 2: Config Compliance Rules Constantly Non-Compliant Due to Terraform Plan Artifacts

The automated compliance gate was blocking pipeline runs because S3 buckets created temporarily during Terraform plan operations weren't encrypted — they existed for less than 3 minutes but Config evaluated them immediately.

How I troubleshot it: CloudTrail showed S3 CreateBucket events followed immediately by DeleteBucket events within 2-3 minutes. Config was capturing the intermediate "non-compliant" state of the short-lived bucket and marking the conformance pack as failed.

Solution: I added a config:EvaluateConfigRuleCompliance wait step with jitter before the compliance gate actually checked results. More importantly, I moved ephemeral Terraform plan operations to a separate AWS account (the Shared Services account) where the compliance rules were scoped to persistent resources only, not transient build artifacts.

Lesson learned: Config's near-real-time evaluation model can create false compliance failures for ephemeral resources. Design your compliance rules to exclude build-time resources or use resource-level suppression tags.

Challenge 3: Session Manager Connectivity Failures in Private Subnets

Three ECS instances in the private data subnet became unreachable via Session Manager during a network configuration change. No SSH fallback existed (by design), so I needed Session Manager to work.

How I troubleshot it: I checked the SSM Agent status via CloudWatch Logs (SSM agent logs its connection status). The logs showed "Failed to connect to service endpoint" — a clear indication of a networking issue rather than an IAM issue.

Solution: I had accidentally removed the com.amazonaws.ap-south-1.ssmmessages VPC endpoint during a security group cleanup. Restoring the endpoint restored connectivity within 2 minutes.

Lesson learned: Maintain a runbook for Session Manager troubleshooting. The three VPC endpoints required for private subnet access (ssm, ssmmessages, ec2messages) should be deployed from Terraform and protected from manual deletion via SCP. Also document the SSM Agent log location before you need it in an emergency.

Challenge 4: KMS Key Policy Locking Out CodeBuild During Cross-Account Deployments

When I extended the pipeline to deploy to the production account (separate AWS account), CodeBuild started failing with "Unable to decrypt artifact" errors despite the KMS key policy appearing correct.

How I troubleshot it: CloudTrail showed kms:Decrypt calls being denied with ExplicitDeny. But the KMS key policy had the CodeBuild role listed. The issue was that the CodeBuild role was in the Shared Services account, but the principal in the key policy was specified without the cross-account ARN format.

Solution: The fix required two changes: (1) updating the KMS key policy to explicitly list the cross-account role ARN (arn:aws:iam::PROD_ACCOUNT_ID:role/CodePipelineDeployRole), and (2) adding the KMS decrypt permission to the IAM role in the production account. Both conditions must be met for cross-account KMS access.

Lesson learned: Always test cross-account KMS scenarios in a non-production environment first. Cross-account KMS access requires changes in both accounts and the failure mode is a silent permission denial that looks identical to a misconfigured key policy in the same account.

Challenge 5: Pipeline Runtime Costs Exceeding Budget During Load Testing

During a load test phase, a developer left a test pipeline triggered by a misconfigured webhook running for 8 hours. It executed 340 pipeline runs, burning approximately $420 in CodeBuild costs in a single day.

Solution: I implemented multiple guardrails: (1) CodePipeline execution throttling using EventBridge rules to cap pipeline executions at 20/hour, (2) AWS Budgets alert at 80% of the daily CodePipeline cost threshold, (3) a Cost Anomaly Detection alert for >50% hour-over-hour increases in CodeBuild spend.

Results and Metrics

After 60 days in production (prior to the PCI-DSS audit), here were the measurable outcomes:

Security Posture Improvements:

Mean time to detect vulnerabilities: 47 days → 4 minutes (99.9% improvement) — findings now appear in Security Hub within minutes of image push
Critical CVEs reaching production: 100% elimination over the 60-day observation period (vs. 3 incidents in the prior 60 days)
Security findings from SAST per week: 12 high/critical findings identified and resolved that would previously have reached production undetected

Pipeline and Development Velocity:

Deployment frequency: 3/week → 12/week (4x improvement) — smaller, safer deploys became the norm
Lead time for changes: 72 hours → 18 hours (75% reduction)
Change failure rate: 22% → 4% (deployments requiring rollback)
Pipeline execution time: 43 minutes → 17 minutes (60% faster due to caching and parallelization)

Compliance:

Automated compliance evidence: 0% → 100% of evidence machine-generated and timestamped
PCI-DSS audit result: Passed (Level 1 certification achieved in week 12)
Config compliance score across 47 rules: 94% (up from unmeasured/~20% estimated)
Manual security review hours per week: 40 hours → 6 hours (85% reduction)

Cost Optimization:

ECR storage costs: $340/month → $38/month (89% reduction via lifecycle policies)
Build time costs: ~$2,800/month → ~$1,100/month (61% reduction via right-sizing and caching)
Eliminated bastion host EC2 costs: $140/month (replaced by Session Manager)
Total DevOps toolchain spend: $7,200/month (within the $8,000 ceiling)

Reliability:

Production availability: 99.4% → 99.97% (change failure rate reduction + faster rollback)
Mean time to recover (MTTR) from incidents: 4.2 hours → 38 minutes (automated rollback triggered by CloudWatch alarms)

Key Takeaways

What worked exceptionally well:

Parallel security scan stages dramatically cut pipeline execution time without reducing coverage — running SAST, Dockerfile linting, and dependency scanning concurrently rather than sequentially was an easy win
Permission boundaries on CI/CD deploy roles gave us IaC automation power (creating IAM roles) without opening privilege escalation paths — this is the single IAM pattern I now apply to every engagement
Config conformance packs as pipeline gates meant compliance was enforced continuously, not just at audit time — the QSA was genuinely surprised and impressed
Structured exception management for CVE suppressions prevented the all-too-common outcome where teams disable security gates because they generate too much noise
Terraform as the single source of truth for all infrastructure meant disaster recovery and account recreation were genuinely fast and reliable

What I'd do differently next time:

Start the multi-account migration before any other work. I did it in parallel with pipeline development, and the account restructuring caused some painful rework of IAM ARNs and KMS key policies.
Implement Amazon Inspector v2 enhanced scanning from day one rather than starting with basic ECR scanning and upgrading later — the migration required updating Config rules and Security Hub integrations mid-project
Build the developer feedback loop earlier. I focused heavily on the security tooling and didn't prioritize the developer experience (IDE plugins for SAST, pre-commit hooks mirroring the pipeline checks) until week 6. Developers would have accepted the pipeline changes more readily with earlier visibility
Use AWS Config Auto-Remediation for low-risk fixes (like adding missing tags) rather than just alerting — it reduces the compliance backlog significantly

Best Practices Discovered:

Fail-closed, not fail-open: When a security check can't run (e.g., SAST tool crashes), the pipeline should fail. It's tempting to fail-open for availability, but you lose all security guarantees
Every security finding needs an SLA: Critical = 24 hours, High = 7 days, Medium = 30 days. Without an SLA, findings pile up and teams start ignoring them
Tag everything at creation, not retroactively: A created-by-pipeline: true tag on every resource makes cost allocation and security investigation dramatically simpler
Test your DR regularly and automatically: A DR plan that isn't tested regularly is a theoretical DR plan. Integrate DR testing into the pipeline schedule
Security Hub as the single pane of glass: Resist the urge to build custom dashboards for individual security tools. Everything flows to Security Hub, everyone looks at Security Hub

Recommendations for Similar Projects:

Before touching any pipeline code, establish your IAM role hierarchy and KMS key structure. Everything else depends on these
Get developer buy-in early by showing how the pipeline catches real bugs in their code, not just theoretical security issues
Budget 20% of your timeline for the compliance evidence documentation — auditors want specifics, and "we have automated compliance" isn't enough without a clear evidence chain
In a regulated environment (PCI-DSS, HIPAA, SOC 2), deploy the AWS Security Hub PCI DSS standard immediately — it maps 100+ Config rules directly to PCI requirements and gives you an audit-ready compliance dashboard out of the box

Tech Stack Summary

Compute: Amazon ECS Fargate (production), ECS Fargate Spot (non-production), CodeBuild (BUILD_GENERAL1_MEDIUM/LARGE)
Storage: Amazon S3 (artifacts, audit reports, flow logs archive, CloudTrail), S3 Intelligent-Tiering (compliance evidence)
Database: Amazon Aurora PostgreSQL 15.4 (Global Database for DR), RDS Proxy, ElastiCache Redis 7
Networking: Amazon VPC (3-tier), VPC Flow Logs, AWS WAF v2, Application Load Balancer, VPC Interface Endpoints, Transit Gateway
Security: AWS KMS (CMKs per service tier), AWS Secrets Manager, IAM with Permission Boundaries, SCPs, Amazon Inspector v2, Amazon GuardDuty (+ Malware Protection), Amazon Macie, AWS Security Hub, AWS Config + Conformance Packs, CloudTrail (+ CloudTrail Lake), AWS Certificate Manager
Monitoring: Amazon CloudWatch (Metrics, Logs, Alarms, Container Insights, Dashboards), AWS X-Ray, Amazon OpenSearch Service (log analytics), Amazon EventBridge, Amazon SNS
CI/CD: AWS CodePipeline V2, AWS CodeBuild, Amazon ECR (Enhanced Scanning), AWS CodeStar Connections (GitHub integration), Amazon Inspector (InspectorScan pipeline action)
IaC Tools: Terraform (primary), AWS CloudFormation (conformance packs, Config rules), Checkov (IaC scanning), Hadolint (Dockerfile linting), Semgrep (SAST), Bandit (Python SAST)

This project showed me — again — that the best security architecture is one developers don't fight. When the pipeline gives developers faster, more reliable feedback on their code and catches issues that would otherwise cause 2 AM incidents, they stop treating security gates as obstacles and start treating them as features. That cultural shift, more than any individual AWS service, was the real outcome of this engagement.

[Boost]

Manish Kumar — Wed, 18 Feb 2026 05:05:49 +0000

Build AWS Architecture Diagrams Using Amazon Q CLI and MCP: A Comprehensive Guide

Manish Kumar ・ Oct 3 '25

#webdev #mcp #aws #architecture

AWS Cloud Case Study: Migrating a Monolithic PHP Application to AWS ECS with RDS

Manish Kumar — Mon, 16 Feb 2026 14:36:23 +0000

Executive Summary

Project Overview

Successfully migrated a legacy monolithic PHP e-commerce application from a single dedicated server to a modern AWS cloud architecture utilizing Amazon ECS (Fargate), RDS MySQL, and supporting services. The project was completed within the 6-week timeline and under the 2,000/month infrastructure budget, delivering immediate business value and establishing a foundation for future growth.

Business Challenge

The client, a mid-sized e-commerce company serving 50,000 daily active users, faced critical limitations with their legacy infrastructure:

Scalability crisis: Black Friday 2023 caused 4 hours of downtime due to the inability to scale horizontally
Deployment risk: Manual deployments with no rollback strategy created operational anxiety
Cost inefficiency: 800/month dedicated server running at 15% average utilization but unable to handle traffic spikes
Security exposure: Running end-of-life PHP 7.2 with no security patches, creating compliance risks
Availability concerns: Single server architecture with no redundancy or disaster recovery capability

The Challenge

I was approached by a mid-sized e-commerce company in late 2024 that was running a legacy PHP application on a single dedicated server—one of those "grew organically over 8 years" situations. Their entire stack lived on one beefy machine: Apache, PHP 7.2, MySQL, Redis for sessions, and about 200GB of product images scattered across the local filesystem. The application itself was a classic monolith built with a custom PHP framework (pre-Laravel days), serving around 50,000 daily active users during normal periods.

The pain points were mounting:

Scalability nightmare: Black Friday 2023 took the site down for 4 hours because they couldn't scale horizontally. Manual vertical scaling meant scheduling downtime, which their business couldn't afford anymore.
Deployment anxiety: Every code deployment required SSH-ing into the production server, running git pull, and hoping nothing broke. No rollback strategy existed beyond "restore from backup and pray".
Cost inefficiency: They were paying 800/month for a dedicated server that sat at 15% CPU utilization most of the time, but would spike to 100% during promotional campaigns.
Security concerns: Running PHP 7.2 meant no security patches, and the compliance team was breathing down their necks about PCI-DSS requirements.
Database bottleneck: A single MySQL instance with no read replicas meant every analytics query slowed down customer-facing transactions.

The CEO gave me a budget of 2,000/month for infrastructure and a 6-week timeline to migrate without disrupting their upcoming summer sale. The constraint was tight—they couldn't afford more than 15 minutes of downtime total, and rollback capability was non-negotiable.

Initial Assessment

I spent the first week doing a deep dive into their architecture. Here's what I discovered during the analysis:

Application architecture findings:

The codebase was about 120,000 lines of custom PHP, with heavy coupling between the presentation layer, business logic, and data access
Session management was handled by Redis, but it was running on the same server (single point of failure)
File uploads went directly to the local disk, creating state that made horizontal scaling impossible
Database queries were scattered throughout the codebase with no ORM—just raw mysqli calls

Performance bottlenecks identified:

Average page load time: 2.8 seconds (ouch)
Database queries per page: averaging 47 queries, with some pages hitting 200+ (classic N+1 problem)
Peak traffic: 850 requests/minute during flash sales
Memory usage: PHP processes were averaging 128MB each, with occasional memory leaks pushing some to 512MB

Stakeholder interviews revealed:

The development team was small (3 developers) and had minimal DevOps experience
They deployed about 15 times per month, always during off-peak hours (2 AM deployments were the norm)
No automated testing existed, so every deployment felt like Russian roulette
The marketing team wanted the ability to scale up predictably for campaigns without involving engineering

Risk factors I flagged:

The MySQL database had no recent performance baseline—I found queries taking 30+ seconds during peak hours
They had backups, but had never actually tested a restore (spoiler: the first test restore failed)
The PHP codebase used deprecated functions that wouldn't work on PHP 8 without modifications
About 15% of the application logic existed as stored procedures in MySQL, creating tight coupling

After running MySQL's slow query log for 48 hours and analyzing their access patterns with New Relic, I realized this wasn't just about lifting and shifting—we needed thoughtful service decoupling even within a containerized monolith approach.

Solution Design

Given the constraints, I decided against a full microservices rewrite. Instead, I designed a "monolith-first" containerization strategy that would give them immediate benefits while creating a foundation for future decomposition.

Architecture decisions and rationale

Compute: AWS ECS with Fargate

I chose ECS over EC2-based containers for several reasons:

The workload had highly variable traffic patterns (70% idle, 30% burst), making Fargate's pay-per-use model more cost-effective than maintaining EC2 instances
Fargate eliminated the operational overhead of managing container hosts—critical given their small team
Built-in integration with ALB and AWS service mesh simplified the networking layer
For their workload (2 vCPU, 4GB RAM per task), Fargate cost ~67/month per continuously running task versus ~30/month for a comparable t3.medium EC2 instance, but the 40% time they weren't running tasks made Fargate 25% cheaper overall

However, I designed the VPC and Task Definitions to be launch-type agnostic. If they grew to need 24/7 high-density workloads, switching to EC2 launch type later would be straightforward.

Database: Amazon RDS MySQL 8.0

Moving from self-managed MySQL to RDS was non-negotiable:

Automated backups with point-in-time recovery (because their backup strategy was... optimistic)
Multi-AZ deployment for 99.95% availability
Read replicas to offload their heavy analytics queries
Automated minor version patching during maintenance windows
Performance Insights to identify query bottlenecks without third-party APM tools

I chose a db.r6g.xlarge instance (4 vCPUs, 32GB RAM) as the primary, costing about 350/month, with two db.r6g.large read replicas at 175/month each for analytics workloads.

Storage: Amazon EFS for shared files

The 200GB of product images needed to be accessible from multiple container tasks. Options I considered:

S3 with CloudFront: Best practice but required application code changes (300+ file operation calls)
EFS: NFS-compatible, could be mounted as a volume in ECS tasks with zero code changes

I went with EFS for the initial migration to minimize risk, with a plan to migrate to S3 in Phase 2. EFS cost about 60/month for their 200GB with infrequent access storage class.

Networking architecture

I designed a VPC with:

Two availability zones for redundancy
Public subnets (for ALB and NAT Gateway)
Private subnets (for ECS tasks and RDS)
Three subnet tiers: 10.0.0.0/20 for public, 10.0.16.0/20 for app private, 10.0.32.0/20 for data private

Security layering:

Application Load Balancer in public subnets terminating SSL
ECS tasks in private subnets with no direct internet access
RDS in isolated data subnets with security groups allowing only ECS task traffic
Secrets Manager for database credentials (no more hardcoded passwords)
IAM task roles following the least privilege principle

Cost vs. performance trade-offs:

The original 800/month dedicated server would be replaced with:

ECS Fargate: ~480/month (assuming 20% average utilization, 5 tasks during peak)
RDS with read replicas: ~700/month
ALB: ~25/month
EFS: ~60/month
NAT Gateway: ~45/month
CloudWatch and other services: ~90/month

Total: ~1,400/month baseline, with room to scale to 2,000 during peak periods. More expensive than the single server, yes, but with 99.95% uptime, auto-scaling, zero maintenance windows, and the ability to handle 10x traffic spikes.

Implementation Journey

Phase 1: Foundation (Week 1)

The first step was creating a rock-solid network foundation. I used Terraform for all infrastructure provisioning because repeatability and disaster recovery were critical.

VPC and networking setup:

I created a VPC with CIDR 10.0.0.0/16, spanning us-east-1a and us-east-1b. The subnet design followed AWS best practices:

# Public subnets for ALB and NAT
public_subnet_a: 10.0.1.0/24
public_subnet_b: 10.0.2.0/24

# Private subnets for ECS tasks
private_app_subnet_a: 10.0.11.0/24
private_app_subnet_b: 10.0.12.0/24

# Isolated subnets for RDS
private_data_subnet_a: 10.0.21.0/24
private_data_subnet_b: 10.0.22.0/24

I deployed NAT Gateways in both AZs for redundancy, though this doubled the cost. In hindsight, a single NAT Gateway would have been fine for their traffic patterns—a lesson learned that cost an extra 45/month.

IAM roles and security baseline:

I created three primary IAM roles:

ECS Task Execution Role: Allowed ECS to pull images from ECR, fetch secrets from Secrets Manager, and write logs to CloudWatch
ECS Task Role: Granted the application permission to access S3 (for future migration), write to CloudWatch Logs, and nothing else
RDS Enhanced Monitoring Role: Enabled Performance Insights

The security group architecture was restrictive:

ALB security group: Allow inbound 443 from 0.0.0.0/0, outbound to ECS security group on port 80
ECS security group: Allow inbound from ALB only, outbound to RDS on 3306 and internet via NAT
RDS security group: Allow inbound from ECS security group only on port 3306

Initial challenge I faced:

During initial testing, ECS tasks couldn't pull images from ECR. After 30 minutes of head-scratching, I realized the VPC endpoints for ECR weren't configured, forcing traffic through the NAT Gateway. Once I added VPC endpoints for ecr.api, ecr.dkr, and s3 (ECR uses S3 behind the scenes), image pulls became both faster and cheaper. This saved about 15/month in NAT Gateway data processing charges.

Phase 2: Core Services (Week 2-3)

Database migration approach:

Zero-downtime migration was the make-or-break requirement. I used this approach:

Baseline export: Took a snapshot of the production MySQL database during low-traffic hours (Sunday 3 AM)
RDS provisioning: Restored snapshot to a new RDS instance, upgraded from MySQL 5.7 to 8.0
Replication setup: Configured binary log replication from on-prem MySQL to RDS using MySQL native replication
Validation period: Ran replication for 5 days, monitoring lag (stayed under 2 seconds)
Cutover: During a 2-minute maintenance window, I:
- Put application in read-only mode
- Verified replication lag was zero
- Updated database connection string to RDS endpoint
- Enabled writes on new database
- Monitored for 15 minutes before declaring success

The entire cutover took 12 minutes of read-only mode, well within the 15-minute downtime budget.

Pro tip: I set up RDS Performance Insights immediately and discovered three queries consuming 60% of database time. A couple of missing indexes later, average query time dropped from 340ms to 45ms.

Application containerization strategy:

Creating the Docker image was straightforward but had nuances:

FROM php:8.1-apache

# Install PHP extensions the app needed
RUN docker-php-ext-install mysqli pdo pdo_mysql opcache

# Copy Apache config for PHP settings
COPY apache-config.conf /etc/apache2/sites-available/000-default.conf

# Copy application code
COPY src/ /var/www/html/

# Set proper permissions
RUN chown -R www-data:www-data /var/www/html

# Enable Apache modules
RUN a2enmod rewrite

EXPOSE 80

The gotcha: Their application used session.save_path pointing to /tmp, which wasn't persistent across container restarts. I updated the PHP configuration to use their existing Redis instance (which I'd already migrated to ElastiCache).

ECS cluster and task definition:

I created an ECS cluster with Container Insights enabled for monitoring:

aws ecs create-cluster \
  --cluster-name production-ecommerce \
  --settings name=containerInsights,value=enabled

The task definition specified:

CPU: 2048 units (2 vCPU)
Memory: 4096 MB (4GB)
Network mode: awsvpc (required for Fargate)
Log driver: awslogs, streaming to CloudWatch Logs
EFS mount: Product images directory mounted at /var/www/html/uploads
Environment variables: Pulled from Secrets Manager for database credentials

Integration points:

The Application Load Balancer was configured with:

HTTPS listener (port 443) with their SSL certificate from ACM
Target group pointing to ECS service, health check on /health.php
Redirect from HTTP (port 80) to HTTPS
Deregistration delay of 30 seconds (important for graceful shutdowns)

One issue I hit: The default health check interval was 30 seconds, which caused false positives during deployments. I tuned it to 10-second intervals with a 3-second timeout and 2 consecutive healthy checks required.

Phase 3: Advanced Features (Week 4)

Monitoring and logging setup:

With Container Insights enabled, I immediately got visibility into cluster, service, and task-level metrics. I created CloudWatch dashboards showing:

ECS CPU and memory utilization per task
ALB request count, latency (p50, p95, p99), and HTTP status codes
RDS connections, CPU, IOPS, and query performance

I set up CloudWatch Alarms for:

ECS CPU > 70% for 5 minutes (triggers auto-scaling)
ALB target response time > 1 second (alerts development team)
RDS CPU > 80% (pages on-call engineer)
ECS task count < 2 (ensures at least two tasks always running)

The application logs were already going to CloudWatch Logs, so I created metric filters to count PHP errors and alert on spikes.

Auto-scaling configuration:

I implemented target tracking scaling policies for the ECS service:

CPU-based scaling: Target 60% average CPU utilization
- Scale out when average CPU > 60% for 3 minutes
- Scale in when average CPU < 60% for 10 minutes (longer cooldown to prevent flapping)
ALB request count scaling: Target 1000 requests per task per minute
- Ensured no single task got overwhelmed during traffic spikes

The scaling policy configuration:

aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --resource-id service/production-ecommerce/web-service \
  --scalable-dimension ecs:service:DesiredCount \
  --min-capacity 2 \
  --max-capacity 10

aws application-autoscaling put-scaling-policy \
  --policy-name cpu-target-tracking \
  --service-namespace ecs \
  --resource-id service/production-ecommerce/web-service \
  --scalable-dimension ecs:service:DesiredCount \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration file://scaling-policy.json

During the first simulated traffic spike (using Apache Bench to generate 10x normal load), the service scaled from 2 tasks to 7 tasks within 4 minutes. Beautiful.

Disaster recovery implementation:

RDS automated backups ran daily with 7-day retention. I also configured:

Manual snapshots before major deployments
Read replica promotion procedure documented (RTO: 5 minutes, RPO: near-zero with synchronous replication)
Cross-region snapshot copies to us-west-2 for true disaster recovery

For the application layer, the ECS task definition was version-controlled in Git. Rolling back a bad deployment was as simple as updating the service to use the previous task definition revision—typically completed in under 3 minutes.

Phase 4: Optimization (Week 5-6)

Performance tuning steps I took:

After running in production for a week, I analyzed the CloudWatch metrics and made several optimizations:

Rightsized task resources: Initial 2vCPU/4GB was overkill. Average CPU was 25%, memory at 1.2GB. I reduced to 1vCPU/2GB, cutting Fargate costs by 50%.
Enabled OPcache aggressively: Modified PHP configuration to cache compiled code for 1 hour. This alone reduced CPU usage by another 20%.
Tuned Apache MaxRequestWorkers: Set to 50 (from default 150) based on actual concurrent connection patterns, reducing memory footprint.
Implemented database connection pooling: Modified the application to reuse database connections across requests instead of creating new connections, reducing RDS connection count from 200 to 40.

Cost optimization measures:

Beyond right-sizing, I implemented:

RDS Reserved Instances: Committed to 1-year reserved instance for the primary database, saving 35% (~120/month)
CloudWatch Logs retention: Set to 30 days instead of indefinite, reducing storage costs
EFS Intelligent-Tiering: Moved to lifecycle policies that transitioned files not accessed in 30 days to Infrequent Access storage class, cutting EFS costs by 40%
Removed redundant NAT Gateway: Consolidated to a single NAT Gateway after proving traffic patterns didn't justify the redundancy cost

Final optimized monthly cost: 1,250, well under the 2,000 budget.

Security hardening:

Post-launch security audit revealed a few items to tighten:

Enabled VPC Flow Logs: Started logging all network traffic for security auditing
Implemented AWS WAF: Added basic rate limiting and SQL injection protection rules on the ALB
Restricted IAM policies: Removed broad S3 permissions from task role that weren't being used
Enabled RDS encryption at rest: Took a snapshot, created new encrypted instance, migrated with zero downtime using the same replication strategy
Configured AWS Config rules: Automated compliance checks for security group rules and public subnet configurations

Technical Specifications

Compute Configuration

ECS Cluster:

Name: production-ecommerce
Launch type: Fargate
Platform version: 1.4.0 (latest)
Container Insights: Enabled
Capacity providers: FARGATE and FARGATE_SPOT (80/20 split for cost optimization)

ECS Task Definition:

Task CPU: 1024 units (1 vCPU)
Task Memory: 2048 MB (2GB)
Network mode: awsvpc
Container image: 637-account-id.dkr.ecr.us-east-1.amazonaws.com/ecommerce-app:latest
Health check: curl -f http://localhost/health.php || exit 1
EFS volume mount: /var/www/html/uploads

ECS Service:

Desired count: 2 (minimum), 10 (maximum)
Deployment type: Rolling update
Deployment circuit breaker: Enabled (automatic rollback on failure)
Load balancer: Application Load Balancer target group
Service auto-scaling: Enabled with target tracking policies

Database Configuration

RDS Primary Instance:

Engine: MySQL 8.0.35
Instance class: db.r6g.xlarge (4 vCPU, 32GB RAM)
Storage: 500GB gp3 (16,000 IOPS, 1000 MB/s throughput)
Multi-AZ: Enabled
Backup retention: 7 days, automated snapshots at 3 AM UTC
Encryption: Enabled (KMS)
Parameter group: Custom with optimized InnoDB settings

RDS Read Replicas (2):

Instance class: db.r6g.large (2 vCPU, 16GB RAM)
Purpose: Analytics and reporting queries
Replication lag monitoring: CloudWatch alarm if lag > 5 seconds

Network Architecture

VPC Design:

CIDR: 10.0.0.0/16
DNS hostnames: Enabled
Availability zones: us-east-1a, us-east-1b

Subnets:

2 public subnets (10.0.1.0/24, 10.0.2.0/24)
2 private application subnets (10.0.11.0/24, 10.0.12.0/24)
2 isolated database subnets (10.0.21.0/24, 10.0.22.0/24)

Routing:

Public subnets: Route to Internet Gateway
Private subnets: Route to NAT Gateway (single, in us-east-1a)
Database subnets: No internet route

Terraform Snippet for VPC

resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "ecommerce-vpc"
    Environment = "production"
  }
}

resource "aws_subnet" "private_app" {
  count             = 2
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${10 + count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "private-app-${count.index + 1}"
    Tier = "application"
  }
}

resource "aws_ecs_cluster" "main" {
  name = "production-ecommerce"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

resource "aws_ecs_service" "web" {
  name            = "web-service"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = aws_subnet.private_app[*].id
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = "php-app"
    container_port   = 80
  }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
}

Challenges and Solutions

Challenge 1: Session Management Across Multiple Containers

Problem: The application stored PHP sessions in local /tmp directory. When ECS spun up multiple tasks, users would randomly lose their sessions when requests hit different containers. Shopping carts were disappearing, and the CEO was getting angry customer emails.

Troubleshooting process:

Initially thought it was a load balancer sticky session issue, spent 2 hours configuring session affinity
Realized through CloudWatch Logs that session IDs were valid but session data was missing
SSH'd into a running container (via ECS Exec feature) and discovered sessions were stored locally

Solution implemented:

Migrated session storage to ElastiCache Redis (t4g.micro, 15/month)
Updated PHP configuration: session.save_handler = redis and session.save_path = "tcp://cache-endpoint:6379"
Tested by deliberately killing containers mid-session—sessions persisted perfectly

Lesson learned: Always externalize state. What seems like a quick fix (local storage) becomes a blocker for horizontal scaling. Now I audit state management in every migration discovery phase.

Challenge 2: Database Connection Exhaustion

Problem: Two weeks post-launch, during a flash sale, the application started throwing "Too many connections" errors. RDS was limited to 400 concurrent connections, and we were hitting that limit despite only 5 ECS tasks running.

How I troubleshot it:

Used RDS Performance Insights to see connection count spiking to 380+ during peak traffic
Ran SHOW PROCESSLIST on the database—found hundreds of sleeping connections
Reviewed application code: discovered database connections weren't being closed properly, and PHP was creating new connections for every request

Solution implemented:

Implemented persistent database connections in PHP (mysqli_connect with persistent flag)
Added connection pooling logic in the application bootstrap
Set MySQL wait_timeout to 300 seconds (down from 28800) to kill idle connections faster
Monitored RDS connection count over a week—stabilized at 40-60 connections

Lesson learned: Connection management is often overlooked in monolithic PHP applications because a single server masks the issue. Containerization exposes these inefficiencies. Always benchmark connection behavior under load before going live.

Challenge 3: EFS Performance Bottleneck

Problem: After migration, page load times for product pages with images were averaging 4.5 seconds—worse than the original dedicated server. CloudWatch showed EFS I/O wait times spiking.

Troubleshooting:

Used CloudWatch EFS metrics to see BurstCreditBalance was at zero during peak hours
Realized EFS throughput in bursting mode was insufficient for their access patterns (200GB meant baseline throughput of only 10 MB/s)
Profiled application and found it was making thousands of file_exists() checks on EFS for every request

Solution implemented:

Short-term fix: Enabled EFS Provisioned Throughput (100 MB/s), added 300/month cost
Long-term solution: Migrated static assets to S3 + CloudFront over next sprint
- Modified upload handler to push to S3 instead of EFS
- Updated image URLs to use CloudFront distribution
- Reduced EFS to only cache and temporary files
Result: Page load times dropped to 1.2 seconds, EFS costs reduced by switching back to bursting mode

Lesson learned: EFS is convenient for lift-and-shift but not optimized for web-facing static content. Always consider the right storage service for the access pattern. S3 + CloudFront is almost always better for static assets in production.

Challenge 4: Auto-Scaling Overreaction

Problem: The auto-scaling configuration initially caused instability. During traffic spikes, ECS would scale from 2 to 10 tasks in 2 minutes, then scale back down to 2 tasks 5 minutes later when load decreased. This thrashing caused customer-facing errors during task spin-up/shutdown.

How I troubleshot it:

Reviewed CloudWatch metrics and saw rapid desired count changes every few minutes
Realized the default scale-in cooldown was too aggressive (60 seconds)
Also discovered the target metric (CPU percentage) was too sensitive to temporary spikes

Solution implemented:

Increased scale-in cooldown to 600 seconds (10 minutes), giving time for sustained load patterns
Changed scale-out cooldown to 180 seconds (3 minutes) to respond quickly to traffic
Added a secondary scaling metric: ALB RequestCountPerTarget, providing more stable signal
Set minimum task count to 3 (instead of 2) during business hours using scheduled scaling

Lesson learned: Auto-scaling is not "set it and forget it." Proper configuration requires understanding traffic patterns and testing under realistic loads. Conservative scale-in policies prevent thrashing while aggressive scale-out policies handle bursts.

Challenge 5: Deployment-Induced Downtime

Problem: Our first production deployment after go-live caused 30 seconds of 502 errors. Users on the checkout flow abandoned carts, and I had to explain the incident to stakeholders.

Troubleshooting process:

Reviewed ALB access logs—saw 502s occurred during task replacement
Discovered the issue: ECS was terminating old tasks before new tasks passed health checks
Application was also not handling SIGTERM gracefully, cutting off in-flight requests

Solution implemented:

Enabled deployment circuit breaker in ECS service (automatically rolls back failed deployments)
Configured rolling deployment with minimum healthy percent of 100%, maximum percent of 200%
Updated application to handle SIGTERM: gracefully finish in-flight requests before shutdown (max 30 seconds)
Increased ALB deregistration delay to 60 seconds, allowing tasks to drain connections
Added pre-stop lifecycle hooks in task definition to flush logs before termination

Lesson learned: Zero-downtime deployments require coordination between application signal handling, load balancer deregistration timing, and ECS deployment configuration. The defaults assume stateless, fast-starting applications—most PHP apps need tuning.

Results and Metrics

After 3 months of operation on the new AWS architecture, here are the quantified outcomes:

Performance improvements:

Average page load time: Reduced from 2.8 seconds to 1.2 seconds (57% improvement)
Time to first byte (TTFB): Improved from 890ms to 240ms (73% improvement)
Database query response time: Average dropped from 340ms to 45ms (87% improvement)
Peak traffic handling: System now handles 8,500 requests/minute (10x original capacity) without degradation

Cost comparison:

Before: 800/month (dedicated server) + 150/month (CDN) + 100/month (monitoring) = 1,050/month baseline
After: 1,250/month AWS infrastructure (all-inclusive)
Cost increase: 19% higher baseline cost
Value delivered: 99.95% uptime vs. 98.2% previously, eliminated 500 emergency scaling costs per event (happened 4 times/year)
True savings: Reduced operational overhead by 15 hours/month (no more manual scaling, patching, or backup management)

Uptime and reliability gains:

Uptime: Improved from 98.2% to 99.94% over 90-day period
Failed deployments: Zero production-impacting incidents (circuit breaker prevented 3 bad deployments from affecting users)
Recovery time: RTO improved from 4 hours to 8 minutes for application issues, 30 minutes for database issues
Zero unplanned downtime in 90 days vs. 3 incidents in previous 90 days

Time-to-market improvements:

Deployment frequency: Increased from 15 deploys/month to 45 deploys/month (3x)
Deployment duration: Reduced from 25 minutes (manual) to 8 minutes (automated rolling update)
Rollback time: Improved from 35 minutes to 3 minutes
Developer confidence: Team now deploys during business hours instead of 2 AM maintenance windows

Team productivity improvements:

On-call incidents: Reduced from 8/month to 2/month (monitoring and auto-healing eliminated most alerts)
Time spent on infrastructure: Reduced from 40 hours/month to 10 hours/month
Mean time to investigate (MTTI): Reduced from 25 minutes to 7 minutes (thanks to CloudWatch Container Insights and centralized logging)

Business impact:

Successfully handled Black Friday 2025 with zero downtime (peak traffic was 12,000 req/min, system auto-scaled to 9 tasks)
Marketing team now schedules flash sales without engineering involvement (auto-scaling handles it)
PCI-DSS compliance achieved (AWS shared responsibility model simplified audit requirements)

Key Takeaways

After delivering this migration, here's what I'd share with anyone doing similar work:

Start with network and security fundamentals: VPC design, security groups, and IAM roles are unglamorous but will save you countless hours later. Get them right before touching application code.
Externalize all state early: Sessions, file uploads, caches—anything that lives on disk must be identified and migrated to external services before containerization. This was our biggest gotcha.
Database migration deserves 30% of project time: We spent 2 weeks on a database migration strategy for a 500GB database. Worth every hour—zero downtime and zero data loss is non-negotiable for production systems.
Right-sizing is iterative: Start with generous resource allocations, monitor for a week, then optimize. We cut our Fargate costs by 50% after the initial week by right-sizing task definitions based on real usage patterns.
Container Insights is worth enabling from day one: The visibility into task-level metrics, correlated with application logs, reduced our mean time to resolution by 70%. It costs about 30/month but saves hours of troubleshooting.
Auto-scaling needs real-world testing: Synthetic load tests don't capture actual traffic patterns. We had to tune auto-scaling policies three times based on production traffic before getting it right.
Deployment strategies matter more than you think: Graceful shutdowns, health check tuning, and deregistration delays prevented customer-facing errors during deployments. This took 3 failed attempts to get right.
Cost optimization is ongoing: Monthly reviews of CloudWatch metrics revealed opportunities to save 30% through reserved instances, intelligent tiering, and removing over-provisioned resources.

What I'd do differently next time:

Migrate to S3 earlier: We should have moved static assets to S3 + CloudFront during the initial migration rather than using EFS as a crutch. The EFS performance issues cost us 2 weeks of firefighting.
Implement feature flags: We deployed the entire migration as a big-bang cutover. Feature flags would have allowed gradual traffic shifting and faster rollback if issues arose.
Load test more aggressively: Our pre-launch load tests were at 2x expected peak traffic. Real Black Friday traffic hit 3x, causing brief issues. Test at 5x to be safe.
Document runbooks earlier: We created operational runbooks after the first incident. Should have documented "what to do when X happens" scenarios before launch.

Tech Stack Summary

Compute:

AWS ECS (Fargate launch type) for container orchestration
Application Load Balancer for traffic distribution and SSL termination
Amazon ECR for Docker image registry

Storage:

Amazon S3 for static assets and backups
Amazon EFS for shared file storage (temporary, migrating to S3)
CloudFront CDN for global content delivery

Database:

Amazon RDS MySQL 8.0 (Multi-AZ) for primary transactional database
RDS Read Replicas (2x) for analytics and reporting queries
Amazon ElastiCache Redis for session storage and application caching

Networking:

Amazon VPC with public/private/isolated subnet architecture
NAT Gateway for outbound internet access from private subnets
VPC Flow Logs for network traffic auditing
AWS WAF for application-layer security (rate limiting, SQL injection protection)

Security:

AWS IAM for access control and task permissions
AWS Secrets Manager for database credentials and API keys
AWS Certificate Manager for SSL/TLS certificates
AWS KMS for encryption key management

Monitoring:

Amazon CloudWatch Container Insights for ECS metrics
CloudWatch Logs for centralized application and infrastructure logging
CloudWatch Alarms for proactive alerting
RDS Performance Insights for database query analysis

IaC Tools:

Terraform for all infrastructure provisioning (VPC, ECS, RDS, ALB)
AWS CLI for operational tasks and troubleshooting
GitHub Actions for CI/CD pipeline (Docker build, ECR push, ECS deployment)
Docker for containerization

This migration transformed a fragile, monolithic application into a resilient, scalable cloud-native architecture while maintaining business continuity. The key wasn't using the fanciest AWS services—it was understanding the constraints, making pragmatic architecture decisions, and executing a phased migration strategy that minimized risk at every step. Three months in, the team is shipping faster, sleeping better, and the business is growing without infrastructure anxiety.

Conclusion

This migration represents a successful transformation from legacy infrastructure to modern cloud-native architecture. The project delivered immediate business value through improved reliability, performance, and operational efficiency while establishing a scalable foundation for future growth. The containerized monolith approach proved to be the right strategy—achieving 99.94% uptime and 10x capacity without the complexity and risk of a full microservices rewrite.

The investment in proper planning, phased execution, and post-launch optimization resulted in a system that not only meets current business needs but provides the architectural flexibility to support the company's growth trajectory over the next 3-5 years.

Project Status: ✅ Complete and in production with ongoing optimization

Business Owner Satisfaction: High - exceeded uptime and performance targets while staying under budget

ROI Timeline: 12 months (accounting for operational efficiency gains and eliminated incident costs)

The AWS Knowledge Gap: What Certifications Don’t Teach About Production

Manish Kumar — Sat, 14 Feb 2026 06:51:08 +0000

After spending years watching AWS beginners struggle with the same preventable mistakes, I've realized that most courses and certifications focus heavily on theory while skipping the messy, real-world lessons you only learn after making costly errors. This guide covers the practical knowledge that separates classroom learners from production-ready cloud engineers.

The Root Account: Your Most Dangerous Asset

Why Root Account Security Matters More Than You Think

Your AWS root account is not just another admin account—it's the master key to your entire cloud infrastructure. Unlike classroom scenarios where you casually log in as root, production environments treat this account like nuclear launch codes.

What Classrooms Skip:

Root accounts can bypass virtually all permission boundaries and service control policies
A compromised root account means complete account takeover with no recovery options
Root credentials should never be used for daily operations, even if you're a solo developer

Practical Implementation:

Enable MFA immediately—use hardware tokens or authenticator apps, never SMS
Create an IAM user with admin permissions for daily work instead of using root
Store root credentials in a physical safe or password manager with restricted access
Use distribution lists (team-security@company.com) instead of personal emails for root account registration
Set up CloudTrail logging before doing anything else to track all root account activities

The Pitfall:
Many beginners create resources with root credentials during testing and forget to document what was created. When something breaks months later, tracing back who created what becomes a nightmare because CloudTrail wasn't enabled from day one.

IAM: The Service Everyone Underestimates

Why IAM Is Your First Security Layer

Identity and Access Management isn't just about creating users—it's about implementing the principle of least privilege at scale. Classrooms teach you to attach "AdministratorAccess" policies to speed through labs, but production systems require surgical precision.

The Real-World Approach:

Start with Zero and Add Incrementally:

Never grant wildcard permissions ("*") or attach AWS managed policies like AdministratorAccess unless absolutely necessary
Use AWS Policy Simulator to test permissions before applying them to production
Implement policy versioning from the beginning—60% of companies experience incidents due to policy misconfigurations

IAM Roles vs. Users—The Critical Distinction:

IAM Users: Permanent credentials for human identities (developers, operators)
IAM Roles: Temporary credentials for services, applications, or cross-account access
Why It Matters: Roles automatically rotate credentials and can't leak long-term keys

Common IAM Pitfalls:

Over-Permissioned Service Roles: Granting Lambda functions full S3 access when they only need read access to one specific bucket
Credential Exposure: Hardcoding AWS access keys in application code or environment variables that get committed to Git
Missing Resource-Based Policies: Forgetting that S3 buckets, KMS keys, and SNS topics have their own policies that can conflict with IAM policies
Temporary Worker Access That Never Expires: Creating IAM users for contractors and forgetting to delete them after projects end—organizations implementing automatic expiration reduce unauthorized access incidents by 40%

How Services Really Use IAM:

When an EC2 instance needs to access S3, you don't SSH in and configure AWS credentials—you attach an IAM role to the instance. The instance then assumes that role and receives temporary credentials automatically. This same pattern applies to:

Lambda functions accessing DynamoDB
ECS tasks reading secrets from Systems Manager Parameter Store
Step Functions orchestrating multiple service calls

The 50% Security Rule:
Analytics reveal that 50% of security incidents trace back to overly permissive IAM settings. The fix? Implement CloudTrail logging, use AWS Access Analyzer to identify unused permissions, and conduct quarterly IAM audits.

The Region Problem: Your First "Where Did Everything Go?" Moment

Why Multi-Region Awareness Is Critical

Classrooms typically work in one region (usually us-east-1), but production reality involves multiple regions, and this trips up nearly every beginner.

The Classic Mistake:
You create an EC2 instance in us-east-1, then switch to ap-south-1 (closer to your location in Delhi) to check on it. The instance isn't there. You panic, thinking CloudFormation failed. But your instance is fine—you're just looking in the wrong region.

Why Regions Matter:

Most AWS resources are region-specific (EC2, RDS, Lambda, VPC)
Some services are global (IAM, Route 53, CloudFront) but configure region-specific resources
Billing metrics only appear in us-east-1 (Northern Virginia) regardless of where resources run
Data transfer between regions incurs significant costs

Practical Region Strategy:

Choose regions based on latency to end users, data residency laws, and service availability
Use resource tagging with region identifiers for multi-region architectures
Set up consolidated CloudTrail trails that log activities across all regions
Build region-switching into your mental checklist when troubleshooting

The Integration Angle:
When designing disaster recovery or high-availability systems, you'll replicate resources across regions. But not everything replicates automatically—Route 53 can route traffic to healthy regions, but you need Lambda functions or custom automation to copy data between regional S3 buckets or RDS read replicas.

VPC Networking: Where Theory Meets Painful Reality

The Networking Layer Nobody Explains Properly

Classrooms show you the default VPC and call it a day. Production engineers spend weeks designing VPC architectures because mistakes here are expensive and difficult to fix.

What Default VPC Hides:

Default VPCs come with public subnets, internet gateways, and permissive route tables already configured
This convenience creates security bad habits—everything you launch is potentially internet-accessible
Production environments use custom VPCs with careful public/private subnet separation

Public vs. Private Subnets—The Real Difference:

Public Subnets:

Route table has a route to an Internet Gateway (0.0.0.0/0 → igw-xxx)
Resources can receive public IPs and communicate directly with the internet
Use cases: Load balancers, bastion hosts, NAT gateways

Private Subnets:

No direct route to Internet Gateway
Instances need NAT Gateway (or NAT Instance) in public subnet to reach internet for updates
Use cases: Application servers, databases, Lambda functions

The Mistake That Costs Money:
Placing RDS databases in public subnets "just to test connectivity". Even if security groups block external access, this configuration violates compliance frameworks and creates unnecessary attack surface.

Subnet Sizing and CIDR Blocks:

Classrooms teach you CIDR notation (/16, /24, etc.) but skip the capacity planning discussion. Here's what matters:

/16 gives you 65,536 IPs (minus 5 AWS-reserved addresses per subnet)
/24 gives you 256 IPs (minus 5 reserved = 251 usable)
AWS reserves first 4 IPs and last IP in every subnet for network, gateway, DNS, and broadcast

Why It Matters:
If you create a VPC with /24 CIDR block, you can't expand it later without recreating the entire VPC. Always plan for growth—use /16 for the VPC, then carve out /24 subnets as needed.

Security Groups vs. NACLs:

This distinction confuses beginners because both control traffic, but they work at different layers:

Feature	Security Groups	Network ACLs
Operates at	Instance level	Subnet level
Statefulness	Stateful (return traffic automatic)	Stateless (must allow both directions)
Rules	Allow rules only	Both allow and deny rules
Evaluation	All rules evaluated	Rules evaluated in order
Default	Deny all inbound, allow all outbound	Allow all traffic

Practical Example:
Your web server in a public subnet needs to serve HTTPS traffic. You configure:

Security group: Allow inbound 443 from 0.0.0.0/0, allow all outbound (return traffic works automatically)
NACL: Allow inbound 443, allow outbound ephemeral ports (1024-65535) for return traffic

Why Both Exist:
Security groups are your primary defense (whitelist specific access). NACLs act as subnet-level firewall for defense-in-depth and can explicitly deny traffic from known malicious IPs.

Service Integration: How AWS Services Actually Talk to Each Other

The Integration Patterns Classrooms Ignore

Courses teach you individual services in isolation—here's how to create an S3 bucket, here's how to launch Lambda—but skip how these services communicate in real architectures.

Synchronous vs. Asynchronous Communication:

Synchronous (Request-Reply Pattern):

API Gateway receives HTTP request → Lambda processes → Returns response immediately
Client waits for complete response before continuing
Use when: User needs immediate feedback (form submission, search query)
Pitfall: Timeout limits (API Gateway: 29 seconds, Lambda: 15 minutes max)

Asynchronous (Message Queue Pattern):

Application writes message to SQS queue → Lambda polls queue and processes later
Client receives acknowledgment immediately, processing happens in background
Use when: Long-running tasks, decoupling producers from consumers
Benefit: If Lambda fails, message remains in queue for retry (up to 14 days)

Event-Driven Architecture:

Modern AWS architectures emit events rather than calling services directly:

S3 bucket upload triggers EventBridge rule → Invokes Lambda to process image
DynamoDB stream captures changes → Lambda updates search index in ElasticSearch
CloudWatch alarm triggers SNS notification → Fan-out to email, SMS, and Lambda for auto-remediation

Why This Matters:
Tight coupling (direct service-to-service calls) creates fragile systems. If the downstream service is down, your entire application breaks. Event-driven patterns with queues and topics provide resilience—services can fail and recover without data loss.

Real-World Integration Example:

E-commerce Order Processing:

User places order → API Gateway + Lambda writes to DynamoDB
DynamoDB stream triggers Lambda → Publishes event to SNS topic
SNS fans out to multiple SQS queues: inventory, shipping, notifications
Each queue has dedicated Lambda consumers processing independently
Step Functions orchestrates long-running workflows (payment → fulfillment → shipping)

This architecture is resilient (queues buffer load spikes), scalable (each component scales independently), and observable (CloudWatch metrics at each integration point).

Common Integration Pitfalls:

Retry Storms: Lambda fails, SQS retries, Lambda fails again—without exponential backoff or dead-letter queues, you burn money on infinite retries
Circular Dependencies: Lambda A writes to DynamoDB → Stream triggers Lambda B → Lambda B writes to same table → Infinite loop
Missing Error Handling: Assuming every API call succeeds without implementing try-catch blocks or Step Functions error states

How to Design Integration Right:

Use SQS queues between services for buffering and retry logic
Implement dead-letter queues to capture failed messages for analysis
Monitor IteratorAge metric for Kinesis streams—high age means consumers can't keep up
Use X-Ray for distributed tracing to debug cross-service issues

Cost Management: The Bill That Ruins Your Month

Why Billing Surprises Happen to Everyone

Classrooms rarely discuss costs because lab accounts have credits. Real accounts charge real money, and beginners regularly receive surprise bills.

The Most Common Cost Mistakes:

Forgetting to Stop Resources:

Leaving EC2 instances running 24/7 when you only needed them for 2-hour testing
Creating RDS databases without stopping them (some instance types can't be stopped)
Provisioning NAT Gateways (\$32/month per gateway plus data transfer fees) when NAT Instances might suffice for dev environments

Over-Provisioning:
Beginners select the largest instance types "just in case," thinking like traditional on-premise capacity planning. AWS charges by the hour—start small (t3.micro, t3.small) and scale up based on actual CloudWatch metrics.

Data Transfer Costs:

Data transfer IN to AWS is free
Data transfer OUT to internet costs \$0.09/GB (first 10TB tier)
Data transfer between availability zones costs \$0.01/GB each direction
Mistake: Placing application servers and databases in different AZs during development—high availability costs money

S3 Storage Class Mismanagement:

Storing infrequently accessed logs in S3 Standard instead of S3 Glacier or Intelligent-Tiering
Not implementing lifecycle policies to automatically transition old data to cheaper storage tiers
Keeping S3 buckets with versioning enabled indefinitely—every version counts toward storage costs

Practical Cost Management Setup:

Immediate Actions (Do These First):

Enable CloudWatch billing alerts in us-east-1 region
Create Cost Budget with thresholds at 50%, 80%, and 90% of monthly limit
Set up SNS notifications to alert entire team, not just one person's email
Tag all resources with project, environment, and owner tags for cost attribution

Ongoing Monitoring:

Use Cost Explorer to identify top spending services weekly
Enable AWS Budgets for per-service cost tracking (EC2, RDS, Data Transfer)
Set CloudWatch alarms for forecasted costs, not just actual spend—get warned before the bill arrives
Review Trusted Advisor recommendations monthly for unused resources

The Cost Optimization Mindset:

Treat dev/test environments as ephemeral—destroy them nightly, recreate in morning
Use reserved instances or savings plans for steady-state production workloads (up to 72% savings)
Implement auto-scaling to match capacity with demand
Store backups in cheaper regions if compliance allows

Infrastructure as Code: Why Clicking in Console Is a Mistake

The Manual vs. Automated Deployment Divide

Classrooms teach you to click through the AWS Console because it's visual and easy to demonstrate. Production engineers rarely touch the console except for troubleshooting.

Why Manual Deployments Fail:

Not Repeatable: You create a perfect VPC setup in console, then need to replicate it in another region—can you remember every subnet, route table, and security group rule?
Not Documented: Six months later, someone asks "why does this security group allow port 3306 from 10.0.0.0/16?"—nobody remembers
Not Version Controlled: You make a change that breaks production—how do you roll back?
Not Auditable: Compliance requires knowing who changed what and when—console changes are hard to track even with CloudTrail

Infrastructure as Code (IaC) Solutions:

Terraform (Multi-Cloud):

Declarative syntax (you define desired state, Terraform figures out how to achieve it)
State management tracks current infrastructure
Modules enable reusable components across projects
Supports AWS, Azure, GCP, and 1000+ providers

CloudFormation (AWS Native):

Deep AWS integration with service-specific features
No state file to manage (AWS manages state internally)
Stack-based deployment with built-in rollback on failure
Free to use (only pay for created resources)

AWS CDK (Developer-Friendly):

Write infrastructure in familiar programming languages (Python, TypeScript, Java)
Synthesizes to CloudFormation templates
Provides high-level constructs with sensible defaults
Best for developers who prefer code over YAML/JSON

The Real Benefit:
You write infrastructure once, test it thoroughly, then deploy identical copies to dev, staging, and production environments. Changes go through code review like application code. Disaster recovery becomes terraform apply with different variables.

Practical IaC Workflow:

Define infrastructure in Terraform/CloudFormation
Commit to Git repository with descriptive commit messages
CI/CD pipeline runs validation and cost estimation
Automated testing in dev environment
Manual approval gate for production
Deploy with full audit trail of who approved and why

Database Choices: Picking the Wrong Data Store

RDS, DynamoDB, or Something Else?

Classrooms present databases as a menu—here's RDS, here's DynamoDB, pick one. Real architects ask: what are your access patterns, consistency requirements, and scale needs?

Relational Databases (RDS):

When to Use:

Complex queries with JOINs across multiple tables
ACID transactions (banking, e-commerce orders)
Existing applications designed for PostgreSQL/MySQL/SQL Server

Common Mistakes:

Running on single-AZ instead of Multi-AZ for production (no automatic failover)
Not enabling automated backups (disabled by default for manually launched instances)
Public accessibility enabled "for testing" and forgotten
Choosing provisioned IOPS without understanding workload needs (expensive)

Cost Optimization:

Use read replicas to offload read-heavy workloads
Schedule automated snapshots during low-traffic windows
Consider Aurora Serverless for variable workloads (scales to zero when idle)

NoSQL (DynamoDB):

When to Use:

Key-value or document data models
Need single-digit millisecond latency at any scale
Unpredictable traffic patterns (auto-scaling built-in)
Serverless architectures with Lambda integration

Common Mistakes:

Designing without understanding partition keys and sort keys (leads to hot partitions)
Provisioning capacity instead of on-demand for dev/test environments
Not using Global Secondary Indexes effectively (forces expensive table scans)
Storing large blobs in DynamoDB instead of S3 references (400KB item size limit)

The Integration Angle:

RDS works with traditional application servers, connection pooling, and ORMs
DynamoDB integrates natively with Lambda, Step Functions, and API Gateway
Use RDS for legacy migrations, DynamoDB for greenfield serverless projects

Monitoring and Logging: The "Set It Up First" Services

Why Observability Can't Be an Afterthought

Classrooms demonstrate services working perfectly. Production systems fail constantly—you need visibility to diagnose problems.

The Three Pillars:

1. CloudTrail (Audit Logging):

Records every API call made in your account (who, what, when, from where)
Essential for security forensics and compliance
Enable on day one with S3 bucket lifecycle policies for cost management
Set up multi-region trails to capture activities across all regions

2. CloudWatch (Metrics and Alarms):

Collects performance metrics (CPU, memory, disk I/O, network)
Custom metrics for application-level monitoring (order count, login failures)
Alarms trigger notifications or auto-remediation actions
Log aggregation from Lambda, EC2, ECS into searchable log groups

3. X-Ray (Distributed Tracing):

Visualizes request flow through distributed systems
Identifies bottlenecks in multi-service architectures
Traces Lambda → DynamoDB → S3 call chains with latency breakdowns
Essential for debugging microservices and serverless applications

What to Monitor First:

Billing metrics (catch cost overruns early)
EC2 CPU and disk usage (identify right-sizing opportunities)
RDS connections and query performance (prevent connection pool exhaustion)
Lambda errors, duration, and throttles (optimize function performance)
S3 request rates (high rates indicate potential API call costs)

Alerting Best Practices:

Set multiple threshold levels (warning at 70%, critical at 90%)
Route alerts to appropriate teams (don't spam everyone with every alarm)
Include actionable information in notifications (runbook links, affected resources)
Test alert delivery regularly (monthly SNS test messages)

Security Beyond IAM: Layered Defense

The Multi-Layer Security Model

Beginners think security = IAM policies. Production systems implement defense-in-depth with multiple security layers.

Network Security:

Private subnets with no internet gateway access for sensitive workloads
Network ACLs to blacklist known malicious IP ranges
VPC Flow Logs to audit all network traffic for forensics
AWS WAF on API Gateway/CloudFront to block SQL injection and XSS attacks

Data Security:

S3 bucket policies with explicit deny for public access
KMS encryption for data at rest (S3, EBS, RDS)
SSL/TLS for data in transit (enforce HTTPS-only policies)
Secrets Manager for database passwords and API keys (never hardcode credentials)

Application Security:

Lambda function environment variables encrypted with KMS
VPC endpoints for AWS service access without traversing internet
Security groups as default-deny firewalls (explicitly allow only required ports)
Regular patching schedules for EC2 instances (use Systems Manager Patch Manager)

Common Security Pitfalls:

Public S3 Buckets: Enable "Block Public Access" at account level unless specific business need
Weak Password Policies: Enforce strong passwords with MFA for all IAM users
Overly Permissive Security Groups: 0.0.0.0/0 on SSH port 22 is an invitation to attackers—restrict to known IP ranges
Missing Encryption: Compliance frameworks require encryption at rest—enable by default for all data stores

The 40% Improvement Rule:
Organizations implementing proper IAM group management and least-privilege policies report 40% productivity improvements and significant security posture enhancements.

The Learning Path Forward

What to Practice Next

Hands-On Project Ideas:

Build a three-tier web application: VPC with public/private subnets, ALB, EC2 auto-scaling group, RDS in private subnet
Create serverless API: API Gateway + Lambda + DynamoDB with proper IAM roles and CloudWatch monitoring
Implement disaster recovery: Multi-region replication with Route 53 failover and automated backups
Cost optimization exercise: Use AWS Cost Explorer to analyze spending, implement tagging strategy, set up budgets and alerts

Essential Skills to Develop:

Read CloudFormation/Terraform documentation—learn to deploy infrastructure as code
Practice IAM policy writing—use Policy Simulator to test before deploying
Master CloudWatch Logs Insights—query logs to debug production issues
Understand VPC design patterns—public/private subnet separation becomes second nature

The Real Difference:
Classroom learners pass certification exams by memorizing service features. Production engineers succeed by understanding why services work certain ways, anticipating failure modes, and implementing resilient patterns from day one.

Final Advice:
Break things in your own AWS account. Set a $20 monthly budget with alerts, then experiment with every service that interests you. The lessons from recovering a failed deployment or debugging a misconfigured security group are worth far more than any tutorial. Document your mistakes, automate your solutions, and build the muscle memory that separates cloud beginners from cloud architects.

The gap between classroom AWS and production AWS is wide, but it's filled with practical knowledge that becomes intuitive through hands-on experience. Start building, start breaking, and start learning the lessons that no classroom can teach.

Accelerate Your AWS Cloud Journey: Comprehensive Resources for Modern Cloud Professionals

Manish Kumar — Sun, 08 Feb 2026 06:53:46 +0000

Are you preparing for AWS certifications, looking to master Infrastructure as Code, or seeking to level up your cloud architecture skills? Whether you're a aspiring cloud engineer or an experienced DevOps professional, having the right learning resources can make all the difference between struggling through documentation and confidently building production-ready solutions.

Why These Resources Stand Out

In today's fast-paced cloud landscape, professionals need more than just theory—they need practical, battle-tested knowledge from someone who's been in the trenches. With over 10 years of IT infrastructure experience and deep expertise in AWS cloud architecture, these guides and templates are designed to help you save time and build smarter.

The resources available at manishpcp.gumroad.com bridge the gap between certification study materials and real-world implementation, giving you the exact tools and knowledge you need to succeed in cloud engineering roles.

What You'll Find

AWS Certification Preparation Materials

Preparing for AWS Solutions Architect Associate or Professional certifications requires more than memorizing services. The comprehensive guides include:

Curated interview question banks covering EC2, ECS, EKS, RDS, S3, Lambda, and more
Real-world scenario-based questions that mirror actual certification exam patterns
Detailed troubleshooting guides for common AWS challenges you'll face in production environments
Step-by-step walkthroughs that explain not just the "what" but the "why" behind AWS best practices

Infrastructure as Code Templates

Stop starting from scratch with every project. Access production-ready templates for:

Terraform configurations for multi-tier AWS architectures
CloudFormation templates for automated infrastructure provisioning
Policy as code implementations for security and compliance automation
Reusable modules that follow AWS Well-Architected Framework principles

These templates aren't just copy-paste solutions—they're educational resources that help you understand the patterns and practices behind scalable cloud infrastructure.

DevOps Automation Scripts

Automate repetitive tasks and streamline your workflows with practical scripts covering:

AWS CLI automation for common operations
Python (Boto3) scripts for AWS resource management
Monitoring and alerting configurations using CloudWatch
Security automation for compliance and best practices enforcement

Migration and Modernization Guides

Moving to the cloud or modernizing existing workloads? Get comprehensive checklists and strategies for:

Large-scale AWS migration planning
Containerization strategies with ECS and EKS
Serverless architecture patterns using Lambda and related services
High availability and disaster recovery implementation guides

Who These Resources Are For

Career Changers and Entry-Level Engineers

Breaking into cloud computing can feel overwhelming with so many services and concepts to learn. These resources provide structured learning paths that take you from foundational concepts to job-ready skills, with clear explanations and practical examples.

Mid-Level Professionals Seeking Advancement

Moving from junior to senior roles requires demonstrating architecture and design skills beyond basic service knowledge. The advanced guides and interview preparation materials help you showcase the expertise needed for Solutions Architect, DevOps Engineer, and Cloud Infrastructure Lead positions.

Technical Interviewers and Hiring Managers

Building effective interview processes for cloud roles requires comprehensive question banks that assess real-world skills. Access curated interview materials that help you evaluate candidates on practical cloud architecture and DevOps capabilities.

Trainers and Content Creators

Creating quality training materials takes significant time and expertise. Leverage professionally developed resources as foundations for your own courses, workshops, or internal team training programs.

The Practical Difference

Unlike generic AWS documentation or basic tutorials, these resources reflect hands-on experience with:

Real production environments requiring security, scalability, and cost optimization
Complex migration projects involving legacy system modernization
Enterprise-grade architectures with compliance and governance requirements
DevOps pipelines integrating Infrastructure as Code with continuous delivery

Every guide, template, and script is crafted with attention to best practices, security considerations, and operational efficiency.

Built by a Cloud Professional, For Cloud Professionals

These resources come from extensive experience designing secure, scalable AWS environments and leading DevOps initiatives. With certifications including AWS Solutions Architect Professional, CloudFormation Master Class, and DevOps specialization, you're learning from someone who has solved the exact challenges you're facing.

The passion for knowledge sharing and driving cloud adoption through automation and best practices shines through every resource. This isn't about selling products—it's about helping fellow cloud professionals accelerate their journey and avoid common pitfalls.

Start Building Smarter Today

Time is your most valuable resource as a cloud professional. Instead of spending weeks piecing together information from scattered documentation and blog posts, access comprehensive, organized resources that give you exactly what you need when you need it.

Whether you're preparing for your next certification, building production infrastructure, or interviewing for a cloud role, having the right guides and templates at your fingertips accelerates your success.

Visit manishpcp.gumroad.com to explore the complete collection of AWS and DevOps resources designed to save you time and help you build smarter.

Invest in Your Cloud Career

The cloud industry continues to grow, with AWS skills remaining among the most in-demand in technology. Every hour you invest in learning cloud architecture, Infrastructure as Code, and DevOps practices compounds your career opportunities and earning potential.

These resources aren't expenses—they're investments in your professional development that pay dividends through faster learning, better job opportunities, and increased confidence in your technical abilities.

Stop struggling with fragmented learning materials and start accessing comprehensive, practical resources created specifically for cloud professionals who want to excel.

Ready to accelerate your AWS journey? Explore the guides, tools, and templates at manishpcp.gumroad.com today.

AWS EC2 Deep Dive: Architecture, Operations, and Best Practices

Manish Kumar — Mon, 19 Jan 2026 06:43:26 +0000

AWS EC2 Complete Working Reference Guide

Instance Types and Families

Instance Type Nomenclature

Format: [Family][Generation][Additional Capabilities].[Size]
Example: c7g.xlarge
- c = Compute optimized family
- 7 = 7th generation
- g = AWS Graviton processor
- xlarge = Size

Instance Families Overview

Family	Category	Processor Options	Use Cases	Key Characteristics
T3, T3a, T4g	General Purpose	Intel, AMD, Graviton	Web servers, dev/test, microservices	Burstable CPU, cost-effective
M5, M6i, M7i	General Purpose	Intel, AMD, Graviton	Databases, application servers	Balanced CPU/memory/network
C5, C6i, C7g	Compute Optimized	Intel, AMD, Graviton	HPC, batch processing, gaming	High CPU-to-memory ratio
R5, R6i, R7g, X1, X2	Memory Optimized	Intel, AMD, Graviton	In-memory databases, big data	High memory-to-CPU ratio
I3, I4i, D2, D3	Storage Optimized	Intel, AMD	Data warehousing, NoSQL, distributed file systems	High IOPS, local NVMe storage
P4, P5, G5, Inf2, Trn1	Accelerated Computing	NVIDIA GPUs, AWS Trainium/Inferentia	ML training/inference, rendering	GPUs, TPUs, specialized accelerators
Mac	General Purpose	Apple Silicon	iOS/macOS development	Dedicated Mac hardware
Hpc7g	HPC Optimized	Graviton	Molecular dynamics, CFD simulations	Optimized for tightly coupled workloads

Instance Sizes

nano, micro, small, medium
large, xlarge, 2xlarge, 4xlarge, 8xlarge, 12xlarge, 16xlarge, 24xlarge, 32xlarge, 48xlarge, 56xlarge, 112xlarge
Each size typically doubles vCPUs and memory from previous size
Metal instances provide access to physical server resources

Processor Variants

Intel: Standard option (M5, C5, R5)
AMD: Cost-optimized (M5a, C5a, R5a - typically 10% cheaper)
AWS Graviton: ARM-based, up to 40% better price-performance (M7g, C7g, R7g)
g suffix: Graviton processor
a suffix: AMD processor
n suffix: Enhanced networking
d suffix: Instance store volumes included

Pricing Models Comparison

Model	Commitment	Savings	Flexibility	Best For	Interruption Risk
On-Demand	None	None (baseline)	Full	Spiky workloads, dev/test	None
Reserved Instances	1-3 years	Up to 72%	Instance family/region locked	Predictable, steady-state workloads	None
Savings Plans - Compute	1-3 years	Up to 66%	Any instance type/region	Flexible compute usage	None
Savings Plans - EC2	1-3 years	Up to 72%	Instance family locked, region locked	Predictable EC2 usage in specific family	None
Spot Instances	None	Up to 90%	Full	Fault-tolerant, batch jobs	Yes (2-minute warning)
Dedicated Hosts	On-demand or 1-3 year	Additional RI discounts	Physical server control	BYOL, compliance	None
Capacity Reservations	On-demand	None (billed if unused)	AZ-specific capacity	Business-critical apps	None

Spot Instance Characteristics

Variable pricing based on supply/demand
2-minute interruption notification
Can be 85% cheaper than On-Demand during low demand periods
Example: c7i.2xlarge at \$0.054/hour (Spot) vs \$0.357/hour (On-Demand)
Best for: Stateless applications, CI/CD, data processing, containerized workloads

Savings Plans Priority

Applies to On-Demand usage first
Leftover commitment applies to Spot at Spot rates
Example: \$100/hour plan with \$80 On-Demand + \$30 Spot = covers \$80 On-Demand fully + \$20 Spot

Reserved Instances Types

Standard RI: Maximum savings, least flexibility
Convertible RI: Can change instance family, lower discount
Scheduled RI: Reserved for specific time windows (deprecated)

Instance Launch Methods

Launch via AWS Console

Navigate to EC2 Dashboard → Launch Instance
Configure:
- Name and tags
- AMI selection (Amazon Linux, Ubuntu, Windows, etc.)
- Instance type
- Key pair (create or select existing)
- Network settings (VPC, subnet, security groups)
- Storage configuration
- Advanced details (user data, IAM role, metadata options)
Review and launch

Launch via AWS CLI

# Basic instance launch
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type t3.medium \
  --key-name MyKeyPair \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0bb1c79de3EXAMPLE \
  --count 1 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=MyInstance}]'

# Launch with user data
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type t3.medium \
  --key-name MyKeyPair \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0bb1c79de3EXAMPLE \
  --user-data file://user-data.sh \
  --iam-instance-profile Name=MyInstanceProfile

# Launch Spot Instance
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type t3.medium \
  --instance-market-options '{"MarketType":"spot","SpotOptions":{"MaxPrice":"0.05","SpotInstanceType":"one-time"}}' \
  --key-name MyKeyPair \
  --security-group-ids sg-0123456789abcdef0

User Data Script Example

#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo "<h1>Hello from $(hostname -f)</h1>" > /var/www/html/index.html

# Get instance metadata
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
AZ=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
echo "<p>Instance ID: $INSTANCE_ID</p>" >> /var/www/html/index.html
echo "<p>Availability Zone: $AZ</p>" >> /var/www/html/index.html

Launch with Terraform

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  key_name      = "MyKeyPair"

  vpc_security_group_ids = [aws_security_group.web.id]
  subnet_id              = aws_subnet.public.id

  iam_instance_profile = aws_iam_instance_profile.ec2_profile.name

  user_data = <<-EOF
              #!/bin/bash
              yum update -y
              yum install -y httpd
              systemctl start httpd
              systemctl enable httpd
              EOF

  root_block_device {
    volume_type = "gp3"
    volume_size = 30
    encrypted   = true
  }

  tags = {
    Name        = "WebServer"
    Environment = "Production"
  }

  monitoring = true
}

Launch with CloudFormation

Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-0c55b159cbfafe1f0
      InstanceType: t3.medium
      KeyName: MyKeyPair
      SecurityGroupIds:
        - !Ref WebSecurityGroup
      SubnetId: !Ref PublicSubnet
      IamInstanceProfile: !Ref EC2InstanceProfile
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          systemctl start httpd
          systemctl enable httpd
      BlockDeviceMappings:
        - DeviceName: /dev/xvda
          Ebs:
            VolumeType: gp3
            VolumeSize: 30
            Encrypted: true
      Tags:
        - Key: Name
          Value: WebServer

Launch Templates

Create Launch Template via CLI

aws ec2 create-launch-template \
  --launch-template-name MyLaunchTemplate \
  --version-description "Version 1" \
  --launch-template-data '{
    "ImageId": "ami-0c55b159cbfafe1f0",
    "InstanceType": "t3.medium",
    "KeyName": "MyKeyPair",
    "SecurityGroupIds": ["sg-0123456789abcdef0"],
    "IamInstanceProfile": {
      "Name": "MyInstanceProfile"
    },
    "BlockDeviceMappings": [{
      "DeviceName": "/dev/xvda",
      "Ebs": {
        "VolumeSize": 30,
        "VolumeType": "gp3",
        "DeleteOnTermination": true,
        "Encrypted": true
      }
    }],
    "Monitoring": {
      "Enabled": true
    },
    "UserData": "IyEvYmluL2Jhc2gKCnl1bSB1cGRhdGUgLXkKeXVtIGluc3RhbGwgLXkgaHR0cGQ="
  }'

Launch Template with Systems Manager Parameter

# Create SSM parameter for AMI ID
aws ssm put-parameter \
  --name "/golden-ami/latest" \
  --value "ami-0c55b159cbfafe1f0" \
  --type "String"

# Create launch template referencing SSM parameter
aws ec2 create-launch-template \
  --launch-template-name MyTemplate \
  --launch-template-data '{
    "ImageId": "resolve:ssm:/golden-ami/latest",
    "InstanceType": "t3.medium"
  }'

Launch Template with Terraform

resource "aws_launch_template" "app" {
  name_prefix   = "app-"
  image_id      = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  key_name      = "MyKeyPair"

  vpc_security_group_ids = [aws_security_group.app.id]

  iam_instance_profile {
    name = aws_iam_instance_profile.app.name
  }

  block_device_mappings {
    device_name = "/dev/xvda"

    ebs {
      volume_size           = 30
      volume_type           = "gp3"
      iops                  = 3000
      throughput            = 125
      delete_on_termination = true
      encrypted             = true
    }
  }

  network_interfaces {
    associate_public_ip_address = true
    delete_on_termination       = true
    security_groups             = [aws_security_group.app.id]
  }

  monitoring {
    enabled = true
  }

  user_data = base64encode(<<-EOF
              #!/bin/bash
              yum update -y
              yum install -y httpd
              systemctl start httpd
              EOF
  )

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name = "AppServer"
    }
  }
}

Launch Instance from Template

aws ec2 run-instances \
  --launch-template LaunchTemplateName=MyLaunchTemplate,Version=1 \
  --count 2 \
  --subnet-id subnet-0bb1c79de3EXAMPLE

Update Launch Template (Create New Version)

aws ec2 create-launch-template-version \
  --launch-template-id lt-0abcd290751193123 \
  --source-version 1 \
  --launch-template-data '{"InstanceType":"t3.large"}'

Storage Options

Storage Type Comparison

Type	Persistence	Performance	Use Case	Backup Method
EBS (gp3)	Yes (network-attached)	3,000-16,000 IOPS	General purpose, boot volumes	EBS Snapshots
EBS (gp2)	Yes (network-attached)	Up to 16,000 IOPS	Legacy general purpose	EBS Snapshots
EBS (io2)	Yes (network-attached)	Up to 64,000 IOPS	High-performance databases	EBS Snapshots
EBS (st1)	Yes (network-attached)	Throughput-optimized	Big data, data warehouses	EBS Snapshots
EBS (sc1)	Yes (network-attached)	Cold HDD, lowest cost	Infrequent access	EBS Snapshots
Instance Store	No (ephemeral)	Very high IOPS	Temporary data, caches	Must use application-level backup

EBS Volume Types Detailed

gp3 (General Purpose SSD)

3,000 IOPS baseline (configurable up to 16,000)
125 MB/s throughput baseline (configurable up to 1,000 MB/s)
Price: \$0.08/GB-month
Independent IOPS and throughput configuration
Recommended for most workloads

gp2 (General Purpose SSD - Legacy)

IOPS scales with volume size (3 IOPS per GB)
Burstable up to 3,000 IOPS for volumes < 1 TB
Throughput: up to 250 MB/s
Use gp3 for new deployments (better value)

io2 Block Express (Provisioned IOPS SSD)

Up to 256,000 IOPS per volume
99.999% durability
Up to 4,000 MB/s throughput
Sub-millisecond latency
Use for critical databases

EBS Volume Operations

# Create EBS volume
aws ec2 create-volume \
  --availability-zone us-east-1a \
  --size 100 \
  --volume-type gp3 \
  --iops 3000 \
  --throughput 125 \
  --encrypted \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=MyVolume}]'

# Attach volume to instance
aws ec2 attach-volume \
  --volume-id vol-0123456789abcdef0 \
  --instance-id i-0123456789abcdef0 \
  --device /dev/sdf

# Modify volume (increase size and IOPS)
aws ec2 modify-volume \
  --volume-id vol-0123456789abcdef0 \
  --size 200 \
  --iops 5000

# Create snapshot
aws ec2 create-snapshot \
  --volume-id vol-0123456789abcdef0 \
  --description "Backup of MyVolume"

# Create volume from snapshot
aws ec2 create-volume \
  --snapshot-id snap-0123456789abcdef0 \
  --availability-zone us-east-1a \
  --volume-type gp3

# Detach volume
aws ec2 detach-volume \
  --volume-id vol-0123456789abcdef0

# Delete volume
aws ec2 delete-volume \
  --volume-id vol-0123456789abcdef0

EBS Snapshot Management

# Create multi-volume snapshot for entire instance
aws ec2 create-snapshots \
  --instance-specification InstanceId=i-0123456789abcdef0 \
  --description "Full instance backup"

# Copy snapshot to another region
aws ec2 copy-snapshot \
  --source-region us-east-1 \
  --source-snapshot-id snap-0123456789abcdef0 \
  --destination-region us-west-2 \
  --description "DR copy"

# Create AMI from instance (includes all attached EBS volumes)
aws ec2 create-image \
  --instance-id i-0123456789abcdef0 \
  --name "MyGoldenImage" \
  --description "Production baseline" \
  --no-reboot

# List snapshots
aws ec2 describe-snapshots \
  --owner-ids self \
  --filters "Name=status,Values=completed"

# Delete snapshot
aws ec2 delete-snapshot \
  --snapshot-id snap-0123456789abcdef0

Instance Store Characteristics

Physically attached to host server
Data lost on instance stop/terminate/hardware failure
Included in instance price (no additional cost)
Very high IOPS (millions)
Available on specific instance types (c5d, m5d, r5d, i3, i4i)

AMI Management

Create Custom AMI

# Create AMI from running instance (with reboot)
aws ec2 create-image \
  --instance-id i-0123456789abcdef0 \
  --name "MyCustomAMI-$(date +%Y%m%d)" \
  --description "Custom application image"

# Create AMI without rebooting
aws ec2 create-image \
  --instance-id i-0123456789abcdef0 \
  --name "MyCustomAMI" \
  --no-reboot

# Register AMI from snapshot
aws ec2 register-image \
  --name "MyAMI" \
  --root-device-name /dev/xvda \
  --block-device-mappings \
    "DeviceName=/dev/xvda,Ebs={SnapshotId=snap-0123456789abcdef0,VolumeType=gp3}"

AMI Operations

# List AMIs owned by you
aws ec2 describe-images \
  --owners self \
  --filters "Name=state,Values=available"

# Copy AMI to another region
aws ec2 copy-image \
  --source-region us-east-1 \
  --source-image-id ami-0123456789abcdef0 \
  --region us-west-2 \
  --name "MyAMI-Copy"

# Share AMI with another account
aws ec2 modify-image-attribute \
  --image-id ami-0123456789abcdef0 \
  --launch-permission "Add=[{UserId=123456789012}]"

# Make AMI public
aws ec2 modify-image-attribute \
  --image-id ami-0123456789abcdef0 \
  --launch-permission "Add=[{Group=all}]"

# Deregister AMI
aws ec2 deregister-image \
  --image-id ami-0123456789abcdef0

AMI User Data

User data NOT stored in AMI
Must specify user data each time launching from AMI
User data embedded in launch templates persists across launches
AMI captures: OS, applications, configurations, attached EBS volume snapshots

Networking Configuration

Security Groups vs Network ACLs

Feature	Security Groups	Network ACLs
Scope	Instance-level	Subnet-level
State	Stateful (return traffic auto-allowed)	Stateless (must explicitly allow return)
Rules	Allow rules only	Allow and Deny rules
Rule Processing	All rules evaluated	Rules evaluated in order
Default Behavior	Deny all inbound, allow all outbound	Default NACL allows all
Assignment	Must be explicitly assigned	Automatically applied to subnet
Rule Limit	60 inbound + 60 outbound per group	20 inbound + 20 outbound per NACL

Security Group Configuration

# Create security group
aws ec2 create-security-group \
  --group-name WebServerSG \
  --description "Security group for web servers" \
  --vpc-id vpc-0123456789abcdef0

# Add inbound rules
aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 80 \
  --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 443 \
  --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 22 \
  --cidr 203.0.113.0/24

# Allow traffic from another security group
aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 3306 \
  --source-group sg-9876543210abcdef0

# Remove rule
aws ec2 revoke-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 22 \
  --cidr 0.0.0.0/0

# Add outbound rule
aws ec2 authorize-security-group-egress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 443 \
  --cidr 0.0.0.0/0

Security Group with Terraform

resource "aws_security_group" "web" {
  name        = "web-server-sg"
  description = "Security group for web servers"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "HTTP from anywhere"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description     = "SSH from bastion"
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    security_groups = [aws_security_group.bastion.id]
  }

  egress {
    description = "All outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "WebServerSG"
  }
}

Network ACL Configuration

# Create Network ACL
aws ec2 create-network-acl \
  --vpc-id vpc-0123456789abcdef0

# Add inbound rule (allow HTTP)
aws ec2 create-network-acl-entry \
  --network-acl-id acl-0123456789abcdef0 \
  --ingress \
  --rule-number 100 \
  --protocol tcp \
  --port-range From=80,To=80 \
  --cidr-block 0.0.0.0/0 \
  --rule-action allow

# Add deny rule (higher priority)
aws ec2 create-network-acl-entry \
  --network-acl-id acl-0123456789abcdef0 \
  --ingress \
  --rule-number 99 \
  --protocol icmp \
  --icmp-type-code Code=-1,Type=-1 \
  --cidr-block 0.0.0.0/0 \
  --rule-action deny

# Add outbound rule for ephemeral ports
aws ec2 create-network-acl-entry \
  --network-acl-id acl-0123456789abcdef0 \
  --egress \
  --rule-number 100 \
  --protocol tcp \
  --port-range From=1024,To=65535 \
  --cidr-block 0.0.0.0/0 \
  --rule-action allow

# Associate NACL with subnet
aws ec2 replace-network-acl-association \
  --association-id aclassoc-0123456789abcdef0 \
  --network-acl-id acl-0123456789abcdef0

Elastic Network Interface (ENI)

# Create ENI with static private IP
aws ec2 create-network-interface \
  --subnet-id subnet-0123456789abcdef0 \
  --description "Primary network interface" \
  --groups sg-0123456789abcdef0 \
  --private-ip-address 10.0.1.10

# Attach ENI to instance
aws ec2 attach-network-interface \
  --network-interface-id eni-0123456789abcdef0 \
  --instance-id i-0123456789abcdef0 \
  --device-index 1

# Assign secondary private IP
aws ec2 assign-private-ip-addresses \
  --network-interface-id eni-0123456789abcdef0 \
  --private-ip-addresses 10.0.1.11 10.0.1.12

# Detach ENI
aws ec2 detach-network-interface \
  --attachment-id eni-attach-0123456789abcdef0

# Delete ENI
aws ec2 delete-network-interface \
  --network-interface-id eni-0123456789abcdef0

Elastic IP (EIP)

# Allocate Elastic IP
aws ec2 allocate-address --domain vpc

# Associate EIP with instance
aws ec2 associate-address \
  --instance-id i-0123456789abcdef0 \
  --allocation-id eipalloc-0123456789abcdef0

# Associate EIP with ENI
aws ec2 associate-address \
  --network-interface-id eni-0123456789abcdef0 \
  --allocation-id eipalloc-0123456789abcdef0

# Disassociate EIP
aws ec2 disassociate-address \
  --association-id eipassoc-0123456789abcdef0

# Release EIP
aws ec2 release-address \
  --allocation-id eipalloc-0123456789abcdef0

Enhanced Networking

SR-IOV (Single Root I/O Virtualization): Higher PPS, lower latency, lower jitter
ENA (Elastic Network Adapter): Up to 100 Gbps, required for current generation instances
Intel 82599 VF: Up to 10 Gbps, legacy instances
Placement Groups: Cluster, Partition, Spread

Placement Groups

# Create cluster placement group (low latency)
aws ec2 create-placement-group \
  --group-name HPC-Cluster \
  --strategy cluster

# Create partition placement group (distributed)
aws ec2 create-placement-group \
  --group-name BigData-Partition \
  --strategy partition \
  --partition-count 7

# Create spread placement group (high availability)
aws ec2 create-placement-group \
  --group-name Critical-Spread \
  --strategy spread

# Launch instance in placement group
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type c5n.18xlarge \
  --placement "GroupName=HPC-Cluster"

Placement Strategy	Max Instances	Use Case	Characteristics
Cluster	Thousands	HPC, low-latency apps	Single AZ, same hardware
Partition	7 partitions per AZ	Distributed systems (Hadoop, Cassandra)	Isolated hardware per partition
Spread	7 instances per AZ	Critical applications	Each instance on separate hardware

Instance Lifecycle Management

Instance States

State	Description	Billing	Operations Allowed
pending	Launching, preparing	Not billed	Wait
running	Instance is running	Billed	Stop, reboot, hibernate, terminate
stopping	Preparing to stop	Not billed	Wait
stopped	Instance shutdown, can restart	Not billed (storage charges apply)	Start, terminate
shutting-down	Preparing to terminate	Not billed	Wait
terminated	Permanently deleted	Not billed	None (cannot restart)
hibernate	RAM saved to EBS, quick restart	Billed during stopping	Start

Instance Operations

# Start instance
aws ec2 start-instances --instance-ids i-0123456789abcdef0

# Stop instance
aws ec2 stop-instances --instance-ids i-0123456789abcdef0

# Reboot instance
aws ec2 reboot-instances --instance-ids i-0123456789abcdef0

# Terminate instance
aws ec2 terminate-instances --instance-ids i-0123456789abcdef0

# Enable termination protection
aws ec2 modify-instance-attribute \
  --instance-id i-0123456789abcdef0 \
  --disable-api-termination

# Disable termination protection
aws ec2 modify-instance-attribute \
  --instance-id i-0123456789abcdef0 \
  --no-disable-api-termination

# Change instance type (must stop first)
aws ec2 stop-instances --instance-ids i-0123456789abcdef0
aws ec2 modify-instance-attribute \
  --instance-id i-0123456789abcdef0 \
  --instance-type "{\"Value\": \"t3.large\"}"
aws ec2 start-instances --instance-ids i-0123456789abcdef0

Hibernation

RAM contents saved to EBS root volume
Must be enabled at launch
Instance resumes with same instance ID and private IP
Faster startup than stop/start
Requirements:
- Supported instance families: C3-C5, M3-M5, R3-R5, T2-T3
- RAM must be < 150 GB
- Root volume must be EBS, encrypted
- Cannot hibernate > 60 days

# Launch instance with hibernation enabled
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type m5.large \
  --hibernation-options Configured=true \
  --block-device-mappings \
    "DeviceName=/dev/xvda,Ebs={VolumeSize=30,Encrypted=true}"

# Hibernate instance
aws ec2 stop-instances \
  --instance-ids i-0123456789abcdef0 \
  --hibernate

Instance Metadata Service (IMDS)

# IMDSv1 (legacy)
curl http://169.254.169.254/latest/meta-data/

# IMDSv2 (token-based, more secure)
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/

# Common metadata endpoints
# Instance ID
curl http://169.254.169.254/latest/meta-data/instance-id

# Availability Zone
curl http://169.254.169.254/latest/meta-data/placement/availability-zone

# IAM role credentials
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE-NAME

# User data
curl http://169.254.169.254/latest/user-data

Enforce IMDSv2

aws ec2 modify-instance-metadata-options \
  --instance-id i-0123456789abcdef0 \
  --http-tokens required \
  --http-put-response-hop-limit 1

Auto Scaling

Auto Scaling Components

Launch Template: Defines instance configuration
Auto Scaling Group (ASG): Manages instance fleet
Scaling Policies: Define when to scale
Load Balancer: Distributes traffic

Create Auto Scaling Group

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name MyASG \
  --launch-template "LaunchTemplateName=MyLaunchTemplate,Version=1" \
  --min-size 2 \
  --max-size 10 \
  --desired-capacity 3 \
  --vpc-zone-identifier "subnet-0123,subnet-4567,subnet-89ab" \
  --target-group-arns "arn:aws:elasticloadbalancing:region:account:targetgroup/my-tg/abc123" \
  --health-check-type ELB \
  --health-check-grace-period 300 \
  --tags "Key=Name,Value=WebServer,PropagateAtLaunch=true"

Auto Scaling with Terraform

resource "aws_autoscaling_group" "web" {
  name                = "web-asg"
  min_size            = 2
  max_size            = 10
  desired_capacity    = 3
  health_check_type   = "ELB"
  health_check_grace_period = 300
  vpc_zone_identifier = aws_subnet.private[*].id
  target_group_arns   = [aws_lb_target_group.web.arn]

  launch_template {
    id      = aws_launch_template.web.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "WebServer"
    propagate_at_launch = true
  }

  enabled_metrics = [
    "GroupDesiredCapacity",
    "GroupInServiceInstances",
    "GroupMinSize",
    "GroupMaxSize"
  ]
}

Scaling Policies

Target Tracking Scaling

aws autoscaling put-scaling-policy \
  --auto-scaling-group-name MyASG \
  --policy-name target-tracking-cpu \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ASGAverageCPUUtilization"
    },
    "TargetValue": 70.0
  }'

Step Scaling

# Create CloudWatch alarm
aws cloudwatch put-metric-alarm \
  --alarm-name high-cpu \
  --alarm-description "Scale up when CPU > 80%" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=AutoScalingGroupName,Value=MyASG

# Create scaling policy
aws autoscaling put-scaling-policy \
  --auto-scaling-group-name MyASG \
  --policy-name scale-up-policy \
  --policy-type StepScaling \
  --adjustment-type ChangeInCapacity \
  --step-adjustments \
    "MetricIntervalLowerBound=0,MetricIntervalUpperBound=10,ScalingAdjustment=1" \
    "MetricIntervalLowerBound=10,ScalingAdjustment=2"

Scheduled Scaling

aws autoscaling put-scheduled-update-group-action \
  --auto-scaling-group-name MyASG \
  --scheduled-action-name ScaleUpMorning \
  --start-time "2026-01-20T08:00:00Z" \
  --recurrence "0 8 * * MON-FRI" \
  --min-size 5 \
  --max-size 20 \
  --desired-capacity 10

Lifecycle Hooks

aws autoscaling put-lifecycle-hook \
  --lifecycle-hook-name instance-launching-hook \
  --auto-scaling-group-name MyASG \
  --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING \
  --default-result CONTINUE \
  --heartbeat-timeout 300 \
  --notification-target-arn arn:aws:sns:region:account:my-topic

Load Balancing

Load Balancer Types

Type	OSI Layer	Protocol	Use Case	Key Features
Application Load Balancer (ALB)	Layer 7	HTTP/HTTPS	Web applications, microservices	Path/host routing, WebSocket, HTTP/2
Network Load Balancer (NLB)	Layer 4	TCP/UDP/TLS	High-performance, low latency	Static IP, millions RPS, preserve source IP
Gateway Load Balancer (GWLB)	Layer 3	IP	Third-party virtual appliances	Traffic inspection, firewall integration
Classic Load Balancer (CLB)	Layer 4/7	TCP/HTTP	Legacy applications	Deprecated for new deployments

Application Load Balancer with Auto Scaling

# Create target group
aws elbv2 create-target-group \
  --name web-tg \
  --protocol HTTP \
  --port 80 \
  --vpc-id vpc-0123456789abcdef0 \
  --health-check-enabled \
  --health-check-protocol HTTP \
  --health-check-path /health \
  --health-check-interval-seconds 30 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 3

# Create ALB
aws elbv2 create-load-balancer \
  --name web-alb \
  --subnets subnet-0123 subnet-4567 \
  --security-groups sg-0123456789abcdef0 \
  --scheme internet-facing \
  --type application

# Create listener
aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:region:account:loadbalancer/app/web-alb/abc123 \
  --protocol HTTP \
  --port 80 \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:account:targetgroup/web-tg/xyz789

# Add HTTPS listener with SSL certificate
aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:region:account:loadbalancer/app/web-alb/abc123 \
  --protocol HTTPS \
  --port 443 \
  --certificates CertificateArn=arn:aws:acm:region:account:certificate/cert-id \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:account:targetgroup/web-tg/xyz789

ALB with Terraform

resource "aws_lb" "web" {
  name               = "web-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = aws_subnet.public[*].id

  enable_deletion_protection = true
  enable_http2              = true
}

resource "aws_lb_target_group" "web" {
  name     = "web-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    path                = "/health"
    protocol            = "HTTP"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 3
  }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.web.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.web.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = aws_acm_certificate.web.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.web.arn
  }
}

# Path-based routing
resource "aws_lb_listener_rule" "api" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 100

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api.arn
  }

  condition {
    path_pattern {
      values = ["/api/*"]
    }
  }
}

ELB Health Checks

Auto Scaling uses health checks to replace unhealthy instances
Health check types:
- EC2: Instance status checks
- ELB: Load balancer health checks (recommended)
Grace period: Time before health checks start after instance launch

Monitoring and CloudWatch

CloudWatch Metrics for EC2

Basic Monitoring (Free, 5-minute intervals)

CPUUtilization
DiskReadOps, DiskWriteOps
DiskReadBytes, DiskWriteBytes
NetworkIn, NetworkOut
NetworkPacketsIn, NetworkPacketsOut
StatusCheckFailed, StatusCheckFailed_Instance, StatusCheckFailed_System

Detailed Monitoring (Paid, 1-minute intervals)

# Enable detailed monitoring
aws ec2 monitor-instances --instance-ids i-0123456789abcdef0

# Disable detailed monitoring
aws ec2 unmonitor-instances --instance-ids i-0123456789abcdef0

Custom Metrics

Memory utilization (not included by default)
Disk space utilization
Application-specific metrics

CloudWatch Agent Installation

# Download and install CloudWatch agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
sudo rpm -U ./amazon-cloudwatch-agent.rpm

# Configure agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

# Start agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

CloudWatch Agent Configuration (JSON)

{
  "metrics": {
    "namespace": "CustomMetrics/EC2",
    "metrics_collected": {
      "mem": {
        "measurement": [
          {"name": "mem_used_percent", "rename": "MemoryUtilization", "unit": "Percent"}
        ],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": [
          {"name": "used_percent", "rename": "DiskUtilization", "unit": "Percent"}
        ],
        "metrics_collection_interval": 60,
        "resources": ["*"]
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/httpd/access_log",
            "log_group_name": "/aws/ec2/httpd",
            "log_stream_name": "{instance_id}/access_log"
          }
        ]
      }
    }
  }
}

CloudWatch Alarms

# Create CPU alarm
aws cloudwatch put-metric-alarm \
  --alarm-name high-cpu-alarm \
  --alarm-description "Alert when CPU exceeds 80%" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --alarm-actions arn:aws:sns:region:account:my-topic

# Create disk space alarm (custom metric)
aws cloudwatch put-metric-alarm \
  --alarm-name high-disk-usage \
  --alarm-description "Alert when disk usage > 80%" \
  --metric-name DiskUtilization \
  --namespace CustomMetrics/EC2 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0,Name=path,Value=/ \
  --alarm-actions arn:aws:sns:region:account:my-topic

# Create alarm with EC2 action (stop instance)
aws cloudwatch put-metric-alarm \
  --alarm-name stop-instance-on-high-cpu \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 3 \
  --threshold 95 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --alarm-actions arn:aws:automate:region:ec2:stop

Alarm Actions

SNS notification
EC2 action: stop, terminate, reboot, recover
Auto Scaling action
Systems Manager action
Lambda function invocation

CloudWatch Logs

# Create log group
aws logs create-log-group --log-group-name /aws/ec2/application

# Set retention policy
aws logs put-retention-policy \
  --log-group-name /aws/ec2/application \
  --retention-in-days 7

# Create metric filter
aws logs put-metric-filter \
  --log-group-name /aws/ec2/application \
  --filter-name ErrorCount \
  --filter-pattern "[ERROR]" \
  --metric-transformations \
    metricName=ApplicationErrors,metricNamespace=CustomApp,metricValue=1

IAM Roles and Instance Profiles

Create IAM Role for EC2

# Create trust policy
cat > ec2-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create IAM role
aws iam create-role \
  --role-name EC2-S3-Access-Role \
  --assume-role-policy-document file://ec2-trust-policy.json

# Attach policy to role
aws iam attach-role-policy \
  --role-name EC2-S3-Access-Role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# Create instance profile
aws iam create-instance-profile \
  --instance-profile-name EC2-S3-Access-Profile

# Add role to instance profile
aws iam add-role-to-instance-profile \
  --instance-profile-name EC2-S3-Access-Profile \
  --role-name EC2-S3-Access-Role

# Attach instance profile to running instance
aws ec2 associate-iam-instance-profile \
  --instance-id i-0123456789abcdef0 \
  --iam-instance-profile Name=EC2-S3-Access-Profile

IAM Role with Terraform

resource "aws_iam_role" "ec2_role" {
  name = "ec2-app-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "s3_access" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
}

resource "aws_iam_instance_profile" "ec2_profile" {
  name = "ec2-app-profile"
  role = aws_iam_role.ec2_role.name
}

resource "aws_instance" "app" {
  ami                  = "ami-0c55b159cbfafe1f0"
  instance_type        = "t3.medium"
  iam_instance_profile = aws_iam_instance_profile.ec2_profile.name
}

Instance Management Commands

List and Describe Instances

# List all instances
aws ec2 describe-instances

# List instances with specific state
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running"

# List instances with specific tag
aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=Production"

# Get instance details in table format
aws ec2 describe-instances \
  --query 'Reservations[*].Instances[*].[InstanceId,InstanceType,State.Name,PrivateIpAddress,PublicIpAddress,Tags[?Key==`Name`].Value|]' \
  --output table

# Get specific instance details
aws ec2 describe-instances \
  --instance-ids i-0123456789abcdef0

# Get instance status
aws ec2 describe-instance-status \
  --instance-ids i-0123456789abcdef0

Tagging Operations

# Create tags
aws ec2 create-tags \
  --resources i-0123456789abcdef0 \
  --tags Key=Name,Value=WebServer Key=Environment,Value=Production

# Delete tags
aws ec2 delete-tags \
  --resources i-0123456789abcdef0 \
  --tags Key=OldTag

Console Access and Troubleshooting

# Get console output
aws ec2 get-console-output \
  --instance-id i-0123456789abcdef0

# Get console screenshot
aws ec2 get-console-screenshot \
  --instance-id i-0123456789abcdef0

# Get password data (Windows)
aws ec2 get-password-data \
  --instance-id i-0123456789abcdef0 \
  --priv-launch-key-file MyKeyPair.pem

Cost Optimization Best Practices

Right-Sizing

Use AWS Compute Optimizer for recommendations
Monitor CloudWatch metrics for actual utilization
Start with burstable instances (T3/T4g) for variable workloads
Use AWS Cost Explorer to identify underutilized instances

Instance Selection

Prefer Graviton instances (T4g, M7g, C7g) for up to 40% better price-performance
Use AMD instances (T3a, M5a, C5a) for 10% cost savings
Consider Spot instances for fault-tolerant workloads (up to 90% savings)
Implement Savings Plans for committed usage (up to 72% savings)

Storage Optimization

Use gp3 instead of gp2 (20% cheaper, better performance)
Delete unused EBS volumes and snapshots
Implement lifecycle policies for snapshot retention
Use S3 for infrequently accessed data

Auto Scaling Configuration

Set appropriate min/max/desired capacity
Use target tracking for dynamic scaling
Implement scheduled scaling for predictable patterns
Configure scale-in protection for long-running tasks

Monitoring and Cleanup

Tag all resources for cost allocation
Set up billing alerts
Regularly review and terminate unused instances
Use AWS Trusted Advisor for optimization recommendations

Security Best Practices

Network Security

Deploy instances in private subnets
Use security groups with least privilege
Implement Network ACLs for subnet-level filtering
Enable VPC Flow Logs for traffic analysis
Use AWS PrivateLink for service access

Access Control

Use IAM roles instead of access keys
Implement least privilege IAM policies
Enable MFA for privileged operations
Use Systems Manager Session Manager instead of SSH (no key management)
Rotate SSH keys regularly

Data Protection

Enable EBS encryption by default
Encrypt snapshots
Use encrypted AMIs
Implement backup strategies
Enable termination protection for critical instances

Instance Hardening

Keep OS and applications updated
Use AWS Systems Manager Patch Manager
Implement host-based firewalls
Disable unnecessary services
Use IMDSv2 for metadata access
Enable CloudWatch Logs for audit trails

Monitoring and Compliance

Enable CloudTrail for API logging
Use AWS Config for compliance monitoring
Implement AWS Security Hub
Set up CloudWatch alarms for security events
Regular security assessments and penetration testing

This comprehensive reference provides all essential working details for AWS EC2 operations in a structured, point-wise format suitable for quick reference and immediate implementation.