AWS & SRE Field Manual (Part 3): Amazon EC2 Architecture, Hardware Strategy & Modern Autoscaling

1. TL;DR & Problem Statement

• Definition: A core cloud compute service providing resizable, secure, and on-demand virtual machines (instances) with customizable hardware architectures, CPU/memory ratios, and operating systems.
• Problem Solved: Eliminates physical server procurement lead times (weeks to months), datacenter maintenance overhead, and capacity planning risks by enabling instant provisioning of scalable infrastructure for containerized workloads, databases, and application backends.
• Category: Compute / Infrastructure-as-a-Service (IaaS)

---

2. Core Architecture & Key Components

                                  ┌───────────────────────────────────────────┐
                                  │            Amazon EC2 Security            │
                                  │           Stateful Firewalls              │
                                  └─────────────────────┬─────────────────────┘
                                                        │
                                                        ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                       Workload Optimization Spectrum                                        │
│                                                                                                             │
│  ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────┐  │
│  │ General Purpose (T/M) │   │ Compute Optimized (C) │   │ Memory Optimized (R)  │   │  GPU / AI (G/P)   │  │
│  │ Burstable / Balanced  │   │ High CPU / Analytics  │   │ In-Memory / Redis     │   │ Machine Learning  │  │
│  └───────────────────────┘   └───────────────────────┘   └───────────────────────┘   └───────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

2.1. Instance Families & Workload Mapping

• General Purpose (T & M Series): Delivers balanced compute, memory, and networking resources. Ideal for web servers, microservices, and development environments.
  • T-Series Burstable Capacity: Operates on a CPU Credit model. Instances accumulate credits during idle periods and burst to full CPU capacity when traffic spikes occur, providing a cost-effective compute base for baseline workloads.
• Compute Optimized (C Series): Engineered for high-performance processors. Tailored for compute-intensive applications such as batch processing, media encoding, high-traffic web APIs, and scientific modeling.
• Memory Optimized (R Series): Provides high RAM-to-vCPU ratios. Designed to handle memory-heavy workloads, including in-memory caches (Redis, Valkey, Memcached), large-scale data processing pipelines (Pandas/Spark), and enterprise relational databases.
• Accelerated Computing / GPU (G & P Series): Equipped with high-performance hardware graphics processors (NVIDIA GPUs). Essential for deep learning model training, generative AI inference, high-end rendering, and complex parallel matrix calculations.
  • Kubernetes Connection: In EKS clusters, G/P series instances are assigned Taints (e.g., gpu-workload=true:NoSchedule) to isolate them, ensuring only dedicated machine learning pods with matching Tolerations deploy onto expensive GPU nodes.

2.2. Purchasing Models & Cost Engineering Strategies

• On-Demand Instances: Billed by the second with zero long-term commitments or upfront costs. This is the most expensive purchasing model, intended primarily for short-term testing, dynamic unpredictable spikes, and initial environment setups.
• Savings Plans & Reserved Instances (RI): Committing to run a specific instance profile or consistent compute usage over a 1-year or 3-year term yields discounts ranging from 30% to 72% compared to On-Demand rates. Ideal for baseline production workloads and database clusters that run non-stop.
• Spot Instances: Allows purchasing spare AWS compute capacity at steep discounts (up to 90% off On-Demand pricing).
  • The Interruption Risk: If AWS requires the capacity back for On-Demand customers, it issues a 2-minute warning notification (Spot Interruption Notice) before forcibly terminating the Spot instance.
  • Kubernetes Node Strategy: Stateless web microservices running in Amazon EKS can be deployed onto Spot instance node groups. When AWS reclaims a Spot node, the EKS control plane uses the 2-minute notice window to gracefully drain node connections (kubectl drain) and reschedule pods onto remaining healthy nodes, dramatically lowering compute overhead.

2.3. Placement Groups (Physical Placement Strategies)

• Cluster: Places instances physically close together inside a single Availability Zone to achieve ultra-low network latency and high packet-per-second throughput (ideal for HPC and distributed ML model training).
• Spread: Strictly places each instance across distinct physical hardware racks with independent power and network paths to minimize correlated hardware failure risks for critical single-node instances.
• Partition: Divides groups of instances into logical partitions across distinct hardware racks, ensuring that large distributed systems (Kafka, Cassandra, HDFS) do not share common rack failures.

2.4. EBS-Optimized & Enhanced Networking (ENA)

• EBS-Optimized Instances: Allocates dedicated, isolated network bandwidth strictly for EBS block storage I/O, preventing storage traffic from competing with general application networking.
• Enhanced Networking (ENA): Uses SR-IOV (Single Root I/O Virtualization) to bypass the hypervisor layer, minimizing CPU utilization and providing up to 100+ Gbps network bandwidth with ultra-low jitter.

---

3. Network Security: Security Groups vs. Network ACLs

Incoming Request (Port 8080)
                             │
                             ▼
              ┌─────────────────────────────┐
              │    EC2 Security Group       │
              │  (Implicit Default Deny)    │
              └──────────────┬──────────────┘
                             │  Matched Ingress Rule?
                             ├─── No  ──► [Dropped / Blocked]
                             │
                             └─── Yes ──► [Allowed to Instance]
                                              │
                                              ▼
                                 Auto-Allowed Return Traffic
                                   (Stateful Mechanism)

• Security Groups (Virtual Firewall):
  • Operates at the virtual network interface (ENI) and instance level.
  • Implicit Default Deny: By default, all incoming traffic is blocked until explicit ingress rules are defined.
  • Stateful Mechanism: Connection tracking is fully stateful. If an inbound request is permitted by an ingress rule, the corresponding outbound response traffic is automatically allowed out of the instance, regardless of outbound/egress rules.
• Network ACLs (NACLs):
  • Operates at the subnet boundary level.
  • Stateless Mechanism: Rules are evaluated sequentially (by rule number) and are completely stateless, requiring explicit allow rules in both inbound and outbound directions (including ephemeral port ranges for return traffic).

---

4. Practical Notes & Configuration Snippets

Secure Metadata and Token Retrieval via IMDSv2

# Step 1: Request a short-lived session token (TTL: 21600 seconds)
TOKEN=$(curl -s -X PUT "[http://169.254.169.254/latest/api/token](http://169.254.169.254/latest/api/token)" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Step 2: Fetch instance role credentials securely using the token
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  [http://169.254.169.254/latest/meta-data/iam/security-credentials/](http://169.254.169.254/latest/meta-data/iam/security-credentials/)

Karpenter NodePool Manifest (Modern Spot and On-Demand Node Autoscaling)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: dynamic-spot-pool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64", "arm64"]
      nodeClassRef:
        name: default-ec2-node-class
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

---

5. Gotchas & Common Pitfalls

• IMDSv1 Vulnerability & SSRF Exploits: IMDSv1 responds to unauthenticated, simple HTTP GET requests. If an application hosted on the instance contains a Server-Side Request Forgery (SSRF) vulnerability, attackers can fetch http://169.254.169.254 to steal the temporary IAM role credentials assigned to the EC2 instance. Enforce IMDSv2 (HttpTokens=required) across all launch templates.
• T-Series CPU Credit Depletion Throttling: When a burstable T-series instance exhausts its CPU credit balance, AWS forcefully throttles CPU execution down to the baseline performance limit (e.g., 10-20% CPU utilization). In production, either migrate to dedicated M or C series instances or enable T Unlimited mode to prevent complete service lockups.
• Stop vs. Terminate Cost Dynamics: Stopping an EC2 instance halts vCPU and RAM compute charges immediately; however, all attached Amazon EBS storage volumes, provisioned IOPS, and allocated Elastic IPs continue to generate ongoing billing costs until explicitly released or terminated.

---

6. Production Best Practices

• Eliminate Inbound SSH (AWS Systems Manager Session Manager): Close inbound port 22 in all Security Groups and remove local .pem key pairs. Use AWS SSM Session Manager to connect to instances via encrypted IAM-authenticated sessions, logging all terminal activity to CloudTrail and Amazon S3.
• Adopt AWS Graviton (ARM64) Instances: Migrate x86 workloads to ARM64 Graviton instances (c7g, m7g, r7g) to reduce direct instance costs by 20% while gaining up to 40% higher price-performance efficiency.
• Deploy Karpenter over Legacy Cluster Autoscaler: Rather than relying on static Auto Scaling Groups that require predefined instance sizes, deploy Karpenter to inspect pending pod specs directly and provision exact-fit Spot/On-Demand instance fleets within seconds.
• Configure Automated EC2 Instance Recovery: Set up a CloudWatch Alarm monitoring the StatusCheckFailed_System metric with an attached EC2 Auto-Recovery Action. If the underlying physical hardware hypervisor crashes, AWS automatically reboots and migrates the instance to a healthy physical host while retaining the same instance ID, IP address, and EBS state.