AWS & SRE Field Manual (Part 4): Amazon EBS Architecture, Volume Performance & Kubernetes State

1. TL;DR & Problem Statement

• Definition: A high-performance, persistent block-level storage service designed for use with Amazon EC2 instances and Kubernetes worker nodes. Unlike ephemeral instance store volumes, data on an EBS volume persists independently of the lifecycle of the attached compute instance.
• Problem Solved: Provides durable, stateful block storage for databases, file systems, and enterprise applications with independent provisioning of capacity, IOPS, and throughput, backed by automated point-in-time incremental snapshots.
• Category: Storage / Persistent Block Storage

---

2. Core Architecture & Key Components

                     ┌──────────────────────────────────────────────┐
                     │           AWS EKS / EC2 Compute              │
                     └──────────────────────┬───────────────────────┘
                                            │
                               Storage Attachment (RWO / Multi-Attach)
                                            │
      ┌─────────────────────────────────────┴─────────────────────────────────────┐
      ▼                                                                           ▼
┌─────────────────────────────────┐                             ┌─────────────────────────────────┐
│     Amazon EBS Volume (gp3)     │                             │        EBS Snapshot Engine      │
│  Independent IOPS & Throughput  │                             │  Block-Level Incremental Backup │
└────────────────┬────────────────┘                             └────────────────┬────────────────┘
                 │                                                               │
                 ▼                                                               ▼
┌─────────────────────────────────┐                             ┌─────────────────────────────────┐
│ Dynamic Kubernetes PVC Binding  │                             │   Stored Durably in Amazon S3   │
└─────────────────────────────────┘                             └─────────────────────────────────┘

2.1. Volume Types & Performance Spectrum

• General Purpose SSD (gp3): The modern cloud standard. Decouples storage volume capacity (GiB) from performance metrics (IOPS and throughput in MB/s), delivering a baseline of 3,000 IOPS and 125 MB/s free with every volume.
• General Purpose SSD (gp2 - Legacy): Couples IOPS linearly to capacity (3 IOPS per GiB). Scaling performance requires over-provisioning unused disk size.
• Provisioned IOPS SSD (io2 Block Express): Sub-millisecond latency SAN-grade storage delivering up to 256,000 IOPS, 4,000 MB/s throughput, and 99.999% durability for mission-critical database engines (Oracle, SAP HANA, Microsoft SQL Server).

2.2. EBS Multi-Attach (Clustered Storage)

• Enables attaching a single Provisioned IOPS (io2/io1) volume concurrently to up to 16 nitro-based EC2 instances within the same Availability Zone.
• Requires a cluster-aware file system (e.g., GFS2, OCFS2) to manage write locks and prevent data corruption.

2.3. EBS Snapshots & Data Durability

• Incremental Block-Level Backups: Snapshots capture only the delta (modified blocks) since the previous snapshot, stored durably inside Amazon S3 across multiple Availability Zones.
• Crash-Consistent vs. Application-Consistent: Snapshots taken on running instances are crash-consistent; freeze I/O or flush database buffers to disk prior to snapshot creation for application consistency.

---

3. Deep Dive Engineering & Architectural Comparison

Attribute	Legacy gp2 Volume	Modern gp3 Volume	Provisioned IOPS io2 Block Express
Baseline Performance	Tied to capacity (3 IOPS/GiB)	Fixed 3,000 IOPS & 125 MB/s	Configured per provisioned IOPS
Performance Scaling	Must expand disk capacity	Scale IOPS up to 16,000 independently	Scale up to 256,000 IOPS
Max Throughput	250 MB/s	1,000 MB/s	4,000 MB/s
Durability SLA	99.8%–99.9%	99.8%–99.9%	99.999%
Multi-Attach Support	No	No	Yes (Up to 16 EC2 nodes in same AZ)
Cost Profile	Expensive due to capacity bloat	Up to 20% cheaper per GiB than gp2	Premium pricing for extreme IOPS

---

4. Advanced Integrations & Kubernetes Binding

Kubernetes volumeBindingMode: WaitForFirstConsumer

• Standard EBS volumes are strictly zonal (locked to a specific AZ such as us-east-1a).
• If volumeBindingMode: Immediate is used, the EBS CSI driver provisions the volume in an arbitrary AZ upon PVC creation. If the pod is subsequently scheduled on a worker node in a different AZ, the pod fails to start with FailedAttachVolume.
• WaitForFirstConsumer delays volume creation until the Kubernetes scheduler assigns the pod to a specific node, ensuring the EBS volume is dynamically provisioned in the exact matching AZ.

Fast Snapshot Restore (FSR) & Pre-Warming (Lazy Loading)

• When restoring an EBS volume from an S3 snapshot, storage blocks are pulled on-demand (lazy loaded) upon first access, causing a temporary latency spike.
• Mitigation: Enable Fast Snapshot Restore (FSR) on the snapshot for instantaneous maximum performance, or execute block-level sequential reads (fio or dd) to pre-warm the volume before routing production traffic.

---

5. Practical Notes & Configuration Snippets

Production Kubernetes StorageClass (gp3 with Delayed Binding)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  iops: "4000"
  throughput: "250"
  encrypted: "true"

Terraform: Provisioning an EBS gp3 Volume with Custom IOPS & Throughput

resource "aws_ebs_volume" "database_data" {
  availability_zone = "us-east-1a"
  size              = 200 # 200 GiB
  type              = "gp3"
  iops              = 5000
  throughput        = 250
  encrypted         = true
  kms_key_id        = aws_kms_key.ebs_encryption_key.arn

  tags = {
    Name        = "prod-postgres-data-vol"
    Environment = "production"
  }
}

Volume Initialization / Pre-Warming via Linux CLI

# Read all blocks sequentially from newly attached raw EBS block device
sudo dd if=/dev/xvdf of=/dev/null bs=1M status=progress

---

6. Gotchas & Common Pitfalls

• Cross-AZ Attachment Impossibility: EBS volumes cannot attach across Availability Zone boundaries. Stateful workloads requiring multi-AZ concurrent file sharing must use Amazon EFS (NFS) or Amazon FSx rather than EBS.
• Dynamic Volume Expansion Limits: While AWS allows online expansion of EBS volume size without downtime, reducing the size of an EBS volume is not supported by the AWS API. To shrink a disk, you must provision a smaller volume and copy filesystems over.
• Volume Modification Rate Limits: AWS enforces a mandatory 6-hour cooldown period between modifications (size, IOPS, throughput) on a single EBS volume.

---

7. Production Best Practices

• Always Migrate Legacy gp2 to gp3: Migrating volumes from gp2 to gp3 is a live, zero-downtime operation using the ModifyVolume API. It instantly yields a 20% baseline cost reduction while decoupling IOPS scaling.
• Enforce Account-Level EBS Encryption by Default: Enable the account-level setting EnableEbsEncryptionByDefault across all AWS regions to guarantee that unencrypted block storage cannot be accidentally provisioned via CLI, Console, or CI/CD pipelines.
• AWS Data Lifecycle Manager (DLM) for Automated Backups: Use Amazon Data Lifecycle Manager to automate snapshot creation schedules, cross-region replication for disaster recovery (DR), and automated retention cleanup policies without writing custom Lambda cron scripts.
• Monitor VolumeQueueLength & VolumeThroughputPercentage: Use CloudWatch metrics to detect storage bottlenecks. A continuously elevated VolumeQueueLength indicates that application I/O requests are queuing due to exhausted IOPS limits.