DEV Community: Enes Guler

AWS & SRE Field Manual (Part 4): Amazon EBS Architecture, Volume Performance & Kubernetes State

Enes Guler — Tue, 01 Sep 2026 08:54:53 +0000

1. TL;DR & Problem Statement

Definition: A high-performance, persistent block-level storage service designed for use with Amazon EC2 instances and Kubernetes worker nodes. Unlike ephemeral instance store volumes, data on an EBS volume persists independently of the lifecycle of the attached compute instance.
Problem Solved: Provides durable, stateful block storage for databases, file systems, and enterprise applications with independent provisioning of capacity, IOPS, and throughput, backed by automated point-in-time incremental snapshots.
Category: Storage / Persistent Block Storage

2. Core Architecture & Key Components

                     ┌──────────────────────────────────────────────┐
                     │           AWS EKS / EC2 Compute              │
                     └──────────────────────┬───────────────────────┘
                                            │
                               Storage Attachment (RWO / Multi-Attach)
                                            │
      ┌─────────────────────────────────────┴─────────────────────────────────────┐
      ▼                                                                           ▼
┌─────────────────────────────────┐                             ┌─────────────────────────────────┐
│     Amazon EBS Volume (gp3)     │                             │        EBS Snapshot Engine      │
│  Independent IOPS & Throughput  │                             │  Block-Level Incremental Backup │
└────────────────┬────────────────┘                             └────────────────┬────────────────┘
                 │                                                               │
                 ▼                                                               ▼
┌─────────────────────────────────┐                             ┌─────────────────────────────────┐
│ Dynamic Kubernetes PVC Binding  │                             │   Stored Durably in Amazon S3   │
└─────────────────────────────────┘                             └─────────────────────────────────┘

2.1. Volume Types & Performance Spectrum

General Purpose SSD (gp3): The modern cloud standard. Decouples storage volume capacity (GiB) from performance metrics (IOPS and throughput in MB/s), delivering a baseline of 3,000 IOPS and 125 MB/s free with every volume.
General Purpose SSD (gp2 - Legacy): Couples IOPS linearly to capacity (3 IOPS per GiB). Scaling performance requires over-provisioning unused disk size.
Provisioned IOPS SSD (io2 Block Express): Sub-millisecond latency SAN-grade storage delivering up to 256,000 IOPS, 4,000 MB/s throughput, and 99.999% durability for mission-critical database engines (Oracle, SAP HANA, Microsoft SQL Server).

2.2. EBS Multi-Attach (Clustered Storage)

Enables attaching a single Provisioned IOPS (io2/io1) volume concurrently to up to 16 nitro-based EC2 instances within the same Availability Zone.
Requires a cluster-aware file system (e.g., GFS2, OCFS2) to manage write locks and prevent data corruption.

2.3. EBS Snapshots & Data Durability

Incremental Block-Level Backups: Snapshots capture only the delta (modified blocks) since the previous snapshot, stored durably inside Amazon S3 across multiple Availability Zones.
Crash-Consistent vs. Application-Consistent: Snapshots taken on running instances are crash-consistent; freeze I/O or flush database buffers to disk prior to snapshot creation for application consistency.

3. Deep Dive Engineering & Architectural Comparison

Attribute	Legacy `gp2` Volume	Modern `gp3` Volume	Provisioned IOPS `io2 Block Express`
Baseline Performance	Tied to capacity (3 IOPS/GiB)	Fixed 3,000 IOPS & 125 MB/s	Configured per provisioned IOPS
Performance Scaling	Must expand disk capacity	Scale IOPS up to 16,000 independently	Scale up to 256,000 IOPS
Max Throughput	250 MB/s	1,000 MB/s	4,000 MB/s
Durability SLA	99.8%–99.9%	99.8%–99.9%	99.999%
Multi-Attach Support	No	No	Yes (Up to 16 EC2 nodes in same AZ)
Cost Profile	Expensive due to capacity bloat	Up to 20% cheaper per GiB than `gp2`	Premium pricing for extreme IOPS

4. Advanced Integrations & Kubernetes Binding

Kubernetes `volumeBindingMode: WaitForFirstConsumer`

Standard EBS volumes are strictly zonal (locked to a specific AZ such as us-east-1a).
If volumeBindingMode: Immediate is used, the EBS CSI driver provisions the volume in an arbitrary AZ upon PVC creation. If the pod is subsequently scheduled on a worker node in a different AZ, the pod fails to start with FailedAttachVolume.
WaitForFirstConsumer delays volume creation until the Kubernetes scheduler assigns the pod to a specific node, ensuring the EBS volume is dynamically provisioned in the exact matching AZ.

Fast Snapshot Restore (FSR) & Pre-Warming (Lazy Loading)

When restoring an EBS volume from an S3 snapshot, storage blocks are pulled on-demand (lazy loaded) upon first access, causing a temporary latency spike.
Mitigation: Enable Fast Snapshot Restore (FSR) on the snapshot for instantaneous maximum performance, or execute block-level sequential reads (fio or dd) to pre-warm the volume before routing production traffic.

5. Practical Notes & Configuration Snippets

Production Kubernetes StorageClass (`gp3` with Delayed Binding)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  iops: "4000"
  throughput: "250"
  encrypted: "true"

Terraform: Provisioning an EBS `gp3` Volume with Custom IOPS & Throughput

resource "aws_ebs_volume" "database_data" {
  availability_zone = "us-east-1a"
  size              = 200 # 200 GiB
  type              = "gp3"
  iops              = 5000
  throughput        = 250
  encrypted         = true
  kms_key_id        = aws_kms_key.ebs_encryption_key.arn

  tags = {
    Name        = "prod-postgres-data-vol"
    Environment = "production"
  }
}

Volume Initialization / Pre-Warming via Linux CLI

# Read all blocks sequentially from newly attached raw EBS block device
sudo dd if=/dev/xvdf of=/dev/null bs=1M status=progress

6. Gotchas & Common Pitfalls

Cross-AZ Attachment Impossibility: EBS volumes cannot attach across Availability Zone boundaries. Stateful workloads requiring multi-AZ concurrent file sharing must use Amazon EFS (NFS) or Amazon FSx rather than EBS.
Dynamic Volume Expansion Limits: While AWS allows online expansion of EBS volume size without downtime, reducing the size of an EBS volume is not supported by the AWS API. To shrink a disk, you must provision a smaller volume and copy filesystems over.
Volume Modification Rate Limits: AWS enforces a mandatory 6-hour cooldown period between modifications (size, IOPS, throughput) on a single EBS volume.

7. Production Best Practices

Always Migrate Legacy gp2 to gp3: Migrating volumes from gp2 to gp3 is a live, zero-downtime operation using the ModifyVolume API. It instantly yields a 20% baseline cost reduction while decoupling IOPS scaling.
Enforce Account-Level EBS Encryption by Default: Enable the account-level setting EnableEbsEncryptionByDefault across all AWS regions to guarantee that unencrypted block storage cannot be accidentally provisioned via CLI, Console, or CI/CD pipelines.
AWS Data Lifecycle Manager (DLM) for Automated Backups: Use Amazon Data Lifecycle Manager to automate snapshot creation schedules, cross-region replication for disaster recovery (DR), and automated retention cleanup policies without writing custom Lambda cron scripts.
Monitor VolumeQueueLength & VolumeThroughputPercentage: Use CloudWatch metrics to detect storage bottlenecks. A continuously elevated VolumeQueueLength indicates that application I/O requests are queuing due to exhausted IOPS limits.

AWS & SRE Field Manual (Part 3): Amazon EC2 Architecture, Hardware Strategy & Modern Autoscaling

Enes Guler — Mon, 31 Aug 2026 07:00:20 +0000

1. TL;DR & Problem Statement

Definition: A core cloud compute service providing resizable, secure, and on-demand virtual machines (instances) with customizable hardware architectures, CPU/memory ratios, and operating systems.
Problem Solved: Eliminates physical server procurement lead times (weeks to months), datacenter maintenance overhead, and capacity planning risks by enabling instant provisioning of scalable infrastructure for containerized workloads, databases, and application backends.
Category: Compute / Infrastructure-as-a-Service (IaaS)

2. Core Architecture & Key Components

                                  ┌───────────────────────────────────────────┐
                                  │            Amazon EC2 Security            │
                                  │           Stateful Firewalls              │
                                  └─────────────────────┬─────────────────────┘
                                                        │
                                                        ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                       Workload Optimization Spectrum                                        │
│                                                                                                             │
│  ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────┐  │
│  │ General Purpose (T/M) │   │ Compute Optimized (C) │   │ Memory Optimized (R)  │   │  GPU / AI (G/P)   │  │
│  │ Burstable / Balanced  │   │ High CPU / Analytics  │   │ In-Memory / Redis     │   │ Machine Learning  │  │
│  └───────────────────────┘   └───────────────────────┘   └───────────────────────┘   └───────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

2.1. Instance Families & Workload Mapping

General Purpose (T & M Series): Delivers balanced compute, memory, and networking resources. Ideal for web servers, microservices, and development environments.
- T-Series Burstable Capacity: Operates on a CPU Credit model. Instances accumulate credits during idle periods and burst to full CPU capacity when traffic spikes occur, providing a cost-effective compute base for baseline workloads.
Compute Optimized (C Series): Engineered for high-performance processors. Tailored for compute-intensive applications such as batch processing, media encoding, high-traffic web APIs, and scientific modeling.
Memory Optimized (R Series): Provides high RAM-to-vCPU ratios. Designed to handle memory-heavy workloads, including in-memory caches (Redis, Valkey, Memcached), large-scale data processing pipelines (Pandas/Spark), and enterprise relational databases.
Accelerated Computing / GPU (G & P Series): Equipped with high-performance hardware graphics processors (NVIDIA GPUs). Essential for deep learning model training, generative AI inference, high-end rendering, and complex parallel matrix calculations.
- Kubernetes Connection: In EKS clusters, G/P series instances are assigned Taints (e.g., gpu-workload=true:NoSchedule) to isolate them, ensuring only dedicated machine learning pods with matching Tolerations deploy onto expensive GPU nodes.

2.2. Purchasing Models & Cost Engineering Strategies

On-Demand Instances: Billed by the second with zero long-term commitments or upfront costs. This is the most expensive purchasing model, intended primarily for short-term testing, dynamic unpredictable spikes, and initial environment setups.
Savings Plans & Reserved Instances (RI): Committing to run a specific instance profile or consistent compute usage over a 1-year or 3-year term yields discounts ranging from 30% to 72% compared to On-Demand rates. Ideal for baseline production workloads and database clusters that run non-stop.
Spot Instances: Allows purchasing spare AWS compute capacity at steep discounts (up to 90% off On-Demand pricing).
- The Interruption Risk: If AWS requires the capacity back for On-Demand customers, it issues a 2-minute warning notification (Spot Interruption Notice) before forcibly terminating the Spot instance.
- Kubernetes Node Strategy: Stateless web microservices running in Amazon EKS can be deployed onto Spot instance node groups. When AWS reclaims a Spot node, the EKS control plane uses the 2-minute notice window to gracefully drain node connections (kubectl drain) and reschedule pods onto remaining healthy nodes, dramatically lowering compute overhead.

2.3. Placement Groups (Physical Placement Strategies)

Cluster: Places instances physically close together inside a single Availability Zone to achieve ultra-low network latency and high packet-per-second throughput (ideal for HPC and distributed ML model training).
Spread: Strictly places each instance across distinct physical hardware racks with independent power and network paths to minimize correlated hardware failure risks for critical single-node instances.
Partition: Divides groups of instances into logical partitions across distinct hardware racks, ensuring that large distributed systems (Kafka, Cassandra, HDFS) do not share common rack failures.

2.4. EBS-Optimized & Enhanced Networking (ENA)

EBS-Optimized Instances: Allocates dedicated, isolated network bandwidth strictly for EBS block storage I/O, preventing storage traffic from competing with general application networking.
Enhanced Networking (ENA): Uses SR-IOV (Single Root I/O Virtualization) to bypass the hypervisor layer, minimizing CPU utilization and providing up to 100+ Gbps network bandwidth with ultra-low jitter.

3. Network Security: Security Groups vs. Network ACLs

Incoming Request (Port 8080)
                             │
                             ▼
              ┌─────────────────────────────┐
              │    EC2 Security Group       │
              │  (Implicit Default Deny)    │
              └──────────────┬──────────────┘
                             │  Matched Ingress Rule?
                             ├─── No  ──► [Dropped / Blocked]
                             │
                             └─── Yes ──► [Allowed to Instance]
                                              │
                                              ▼
                                 Auto-Allowed Return Traffic
                                   (Stateful Mechanism)

Security Groups (Virtual Firewall):
- Operates at the virtual network interface (ENI) and instance level.
- Implicit Default Deny: By default, all incoming traffic is blocked until explicit ingress rules are defined.
- Stateful Mechanism: Connection tracking is fully stateful. If an inbound request is permitted by an ingress rule, the corresponding outbound response traffic is automatically allowed out of the instance, regardless of outbound/egress rules.
Network ACLs (NACLs):
- Operates at the subnet boundary level.
- Stateless Mechanism: Rules are evaluated sequentially (by rule number) and are completely stateless, requiring explicit allow rules in both inbound and outbound directions (including ephemeral port ranges for return traffic).

4. Practical Notes & Configuration Snippets

Secure Metadata and Token Retrieval via IMDSv2

# Step 1: Request a short-lived session token (TTL: 21600 seconds)
TOKEN=$(curl -s -X PUT "[http://169.254.169.254/latest/api/token](http://169.254.169.254/latest/api/token)" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Step 2: Fetch instance role credentials securely using the token
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  [http://169.254.169.254/latest/meta-data/iam/security-credentials/](http://169.254.169.254/latest/meta-data/iam/security-credentials/)

Karpenter NodePool Manifest (Modern Spot and On-Demand Node Autoscaling)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: dynamic-spot-pool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64", "arm64"]
      nodeClassRef:
        name: default-ec2-node-class
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

5. Gotchas & Common Pitfalls

IMDSv1 Vulnerability & SSRF Exploits: IMDSv1 responds to unauthenticated, simple HTTP GET requests. If an application hosted on the instance contains a Server-Side Request Forgery (SSRF) vulnerability, attackers can fetch http://169.254.169.254 to steal the temporary IAM role credentials assigned to the EC2 instance. Enforce IMDSv2 (HttpTokens=required) across all launch templates.
T-Series CPU Credit Depletion Throttling: When a burstable T-series instance exhausts its CPU credit balance, AWS forcefully throttles CPU execution down to the baseline performance limit (e.g., 10-20% CPU utilization). In production, either migrate to dedicated M or C series instances or enable T Unlimited mode to prevent complete service lockups.
Stop vs. Terminate Cost Dynamics: Stopping an EC2 instance halts vCPU and RAM compute charges immediately; however, all attached Amazon EBS storage volumes, provisioned IOPS, and allocated Elastic IPs continue to generate ongoing billing costs until explicitly released or terminated.

6. Production Best Practices

Eliminate Inbound SSH (AWS Systems Manager Session Manager): Close inbound port 22 in all Security Groups and remove local .pem key pairs. Use AWS SSM Session Manager to connect to instances via encrypted IAM-authenticated sessions, logging all terminal activity to CloudTrail and Amazon S3.
Adopt AWS Graviton (ARM64) Instances: Migrate x86 workloads to ARM64 Graviton instances (c7g, m7g, r7g) to reduce direct instance costs by 20% while gaining up to 40% higher price-performance efficiency.
Deploy Karpenter over Legacy Cluster Autoscaler: Rather than relying on static Auto Scaling Groups that require predefined instance sizes, deploy Karpenter to inspect pending pod specs directly and provision exact-fit Spot/On-Demand instance fleets within seconds.
Configure Automated EC2 Instance Recovery: Set up a CloudWatch Alarm monitoring the StatusCheckFailed_System metric with an attached EC2 Auto-Recovery Action. If the underlying physical hardware hypervisor crashes, AWS automatically reboots and migrates the instance to a healthy physical host while retaining the same instance ID, IP address, and EBS state.

AWS & SRE Field Manual (Part 2): Application (ALB) vs. Network (NLB) Load Balancing Architecture

Enes Guler — Sun, 30 Aug 2026 09:02:10 +0000

1. TL;DR & Problem Statement

Definition: A fully managed, highly available traffic distribution service that automatically routes incoming application and network traffic across multiple targets (Amazon EC2 instances, EKS Pods, AWS Lambda functions, and IP addresses) across multiple Availability Zones.
Problem Solved: Eliminates single-point-of-failure bottlenecks, handles automated health checks, offloads cryptographic TLS termination, and provides scalable ingress routing mechanisms for cloud-native microservices and high-throughput TCP/UDP streams.
Category: Networking & Content Delivery / Ingress Architecture

2. Core Architecture & Key Components

                                Incoming Client Traffic
                                           │
           ┌───────────────────────────────┴───────────────────────────────┐
           ▼                                                               ▼
┌─────────────────────────────────────────┐             ┌─────────────────────────────────────────┐
│ Layer 7: Application Load Balancer (ALB)│             │  Layer 4: Network Load Balancer (NLB)   │
├─────────────────────────────────────────┤             ├─────────────────────────────────────────┤
│ • Inspects HTTP/HTTPS Headers & Paths   │             │ • Operates strictly at TCP/UDP/TLS      │
│ • Path-Based & Host-Based Routing       │             │ • Non-Inspecting Packet Forwarding      │
│ • SSL Termination & Sticky Sessions     │             │ • Ultra-Low Latency (Sub-millisecond)   │
└────────────────────┬────────────────────┘             └────────────────────┬────────────────────┘
                     │                                                       │
        HTTP / HTTPS Ingress                                    Raw TCP/UDP Stream
                     │                                                       │
        ┌────────────┴────────────┐                                          │
        ▼                         ▼                                          ▼
┌──────────────┐          ┌──────────────┐                        ┌────────────────────┐
│ API Pods     │          │ Frontend Pods│                        │ Ingress Controller │
│ (/api/*)     │          │ (/static/*)  │                        │ / Kafka / Games    │
└──────────────┘          └──────────────┘                        └────────────────────┘

2.1. Application Load Balancer (ALB — OSI Layer 7)

Deep Packet Inspection: Evaluates application-layer headers, hostnames, HTTP methods, query parameters, and URL paths.
Content-Based Routing: Routes traffic intelligently across multiple Target Groups (e.g., /api/* to backend microservices, /static/* to object caches or static web pods).
Modern Protocol Support: Natively supports HTTP/2, gRPC, and WebSockets alongside automated HTTP-to-HTTPS redirect rules.
Sticky Sessions (Cookie Affinity): Binds subsequent requests from a specific client to the same backend target instance/pod using encrypted cookies when state is not externalized.

2.2. Network Load Balancer (NLB — OSI Layer 4)

Raw Transport Passthrough: Operates at the transport layer, routing raw TCP, UDP, and TLS connections based exclusively on IP addresses and ports without payload inspection.
Ultra-Low Latency: Delivers sub-millisecond connection handling directly in the data path.
Instantaneous Burst Capacity: Capable of handling millions of requests per second (RPS) and absorbing sudden, massive traffic spikes without requiring manual pre-warming tickets.
Static & Elastic IPs: Provides one static public IP per Availability Zone, making it ideal for client firewalls requiring strict IP whitelisting.

3. Deep Dive Engineering & Architectural Comparison

Feature / Metric	Application Load Balancer (ALB)	Network Load Balancer (NLB)
OSI Layer	Layer 7 (Application)	Layer 4 (Transport)
Protocols	HTTP, HTTPS, HTTP/2, gRPC, WebSockets	TCP, UDP, TLS
Routing Decisions	URL path, Host header, Query params, HTTP method	Source/Destination IP and Port
Latency Profile	Milliseconds (2-10 ms)	Sub-millisecond (< 1 ms)
Traffic Spikes	Scales elastically via DNS over minutes	Instantaneous line-rate scaling
Static IP Support	Dynamic DNS resolution (CNAME required)	Static Elastic IP per Availability Zone
Client IP Identification	Injects `X-Forwarded-For` HTTP header	Client IP Preservation / Proxy Protocol v2
Primary Use Cases	Microservices routing, REST/gRPC APIs, Web apps	High-throughput streaming, Kafka, Game servers, K8s Ingress Entry

4. Advanced Integrations & Ingress Mechanics

AWS Load Balancer Controller (Instance Mode vs. IP Mode)

Instance Mode (target-type: instance): Routes traffic to EC2 host NodePorts. Requires an internal network hop through kube-proxy and iptables/IPVS, adding latency and SNAT overhead.
IP Mode (target-type: ip): Leverages the AWS VPC CNI to route ingress traffic directly from the ALB/NLB to the individual Kubernetes Pod IP. Bypasses kube-proxy entirely, reducing latency and packet manipulation.

Preserving Real Client IP

ALB: Appends the originating client IP to the X-Forwarded-For and X-Forwarded-Proto request headers.
NLB: Because Layer 4 cannot modify HTTP headers, real client IPs are preserved via native Client IP Preservation or by enabling Proxy Protocol v2 (prepends a binary connection header).

Deregistration Delay (Connection Draining)

Prevents inflight HTTP requests from dropping (502 Bad Gateway) when a backend instance or Kubernetes pod is marked for termination.
The load balancer stops sending new connections to the deregistering target and waits for active transactions to complete within deregistration_delay.timeout_seconds (default: 300s, typically tuned to 15–30s in Kubernetes).

Cross-Zone Load Balancing

Enabled by default on ALB; optional on NLB.
Evenly distributes traffic across all registered targets in all enabled Availability Zones regardless of which AZ received the initial network packet, eliminating uneven target group saturation.

5. Practical Notes & Configuration Snippets

Kubernetes Ingress Manifest (ALB in IP Mode with SSL Redirect)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/abc-123
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=20
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /v1
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 8080

Terraform: Network Load Balancer with Cross-Zone Load Balancing

resource "aws_lb" "network_lb" {
  name                             = "prod-streaming-nlb"
  internal                         = false
  load_balancer_type               = "network"
  subnets                          = var.public_subnet_ids
  enable_cross_zone_load_balancing = true
  enable_deletion_protection       = true
}

resource "aws_lb_target_group" "nlb_tg" {
  name        = "tcp-stream-tg"
  port        = 9092
  protocol    = "TCP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  proxy_protocol_v2 = true # Enables Proxy Protocol v2 for client IP pass-through

  health_check {
    protocol            = "TCP"
    port                = "9092"
    interval            = 10
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }
}

6. Gotchas & Common Pitfalls

Subnet Capacity & IP Exhaustion: Each ALB node placed in a public subnet dynamically scales and consumes multiple private IP addresses within that subnet. If your public subnet CIDR is too narrow (e.g., /28), ALB scale-out events will fail under heavy traffic.
Misconfigured Security Group on NLB: By default, NLBs historically did not have associated Security Groups; firewalling occurred strictly at the backend instance level. While security groups on NLBs are now supported, failing to allow client ingress on the backend node security group when using client IP preservation will cause silent packet drops.
Target Group Health Check Latency: If health check intervals and thresholds are set too conservatively (e.g., 30s interval, 5 unhealthy thresholds), dead or hung pods will receive live user traffic for over 2 minutes before the load balancer drops them from the target rotation.

7. Production Best Practices

AWS WAF Integration on ALB: Always attach an AWS WAF (Web Application Firewall) WebACL to public-facing ALBs to block common OWASP Top 10 vulnerabilities, SQL injection, cross-site scripting (XSS), and rate-limit abusive IP addresses at the cloud edge.
Access Logs to Amazon S3: Enable access logging on both ALB and NLB. Load balancers stream structured access logs (client IP, request processing latency, TLS ciphers, backend response codes) directly to an S3 bucket with lifecycle rules for compliance and security forensics.
Match Ingress Controller Pod Disruption Budgets (PDB): Ensure the deregistration_delay on the AWS Load Balancer matches the terminationGracePeriodSeconds and preStop sleep hook of your Kubernetes pods to guarantee zero-downtime rolling deployments.

AWS & SRE Field Manual (Part 1): Zero-Trust Identity & Access Management (IAM)

Enes Guler — Sat, 29 Aug 2026 08:53:29 +0000

📌 TL;DR & Problem Statement

Definition: The centralized security control plane that manages authentication and authorization across all AWS cloud resources. It serves as the native AWS equivalent of Kubernetes RBAC.
Problem Solved: Secures access to infrastructure resources using centralized, auditable, fine-grained, and temporary security credentials instead of distributed, static passwords and access keys.
Category: Security, Identity, & Compliance
Core Rule (Implicit Default Deny): All API requests across AWS are strictly denied by default unless explicitly granted by a matching Allow statement.

🏛️ Core Architecture & Key Components

1. IAM Users (Human Identities & Static Credentials)

Represents discrete individuals interacting with AWS.
Access is provided via console password with mandatory Multi-Factor Authentication (MFA), or programmatic access via static AccessKeyId and SecretAccessKey pairs.
Root Account: The initial, all-powerful identity created upon account registration. It must be locked behind hardware/virtual MFA immediately, stripped of static access keys, and reserved exclusively for emergency break-glass procedures.

2. IAM Groups (Role-Based Membership)

Logical collections of users sharing identical permission requirements (e.g., DevOps-Admins, Data-Engineers).
Attaching permissions directly to individual users is an anti-pattern; policies attach to groups, and users inherit privileges through group membership.

3. IAM Roles & AWS STS (Temporary Credentials & Workload Identity)

An identity abstraction without permanent passwords or static keys, assumed dynamically via the sts:AssumeRole API.
Mirrors the mechanics of Kubernetes ServiceAccounts and EKS IRSA / Pod Identities. Workloads (EC2 instances, Lambda functions, ECS/EKS pods) obtain short-lived, self-expiring tokens generated by the AWS Security Token Service (STS).
Blast Radius Reduction: Even if a compute instance or container is compromised, the leaked STS token expires within minutes to hours, preventing long-term credential exfiltration.

4. IAM Policies (Declarative Permission Documents)

Structured JSON documents defining who can execute specific actions against target resources under designated conditions:
- Effect: Explicit Allow or Deny (an Explicit Deny unconditionally overrides any Allow).
- Action: Targeted API operations (e.g., s3:GetObject, ec2:DescribeInstances).
- Resource: The target Amazon Resource Name (ARN) governed by the statement (e.g., arn:aws:s3:::production-app-bucket/*).
- Condition: Contextual prerequisites required to satisfy the policy (source IP ranges, MFA enforcement, secure transport, resource/principal tags).

⚙️ Deep Dive Engineering & Authorization Mechanisms

Policy Evaluation Logic (Order of Precedence)

Incoming API Request
        │
        ▼
[ Is there an Explicit Deny? ] ─────── YES ────► [ Access Denied ]
        │
        NO
        ▼
[ Does SCP allow it? ] ─────────────── NO  ────► [ Access Denied ]
        │
       YES
        ▼
[ Does Resource Policy allow it? ] ─── YES ──┐
        │                                    │
        NO                                   │
        ▼                                    │
[ Does Identity Policy allow it? ] ─── YES ──┼──► [ Check Boundaries/Session ] ──► [ Access Granted ]
        │                                    │
        NO                                   │
        ▼                                    │
 [ Access Denied (Implicit Deny) ] ◄─────────┘

Is there an Explicit Deny anywhere in the evaluation chain? (If yes, immediate termination and rejection).
Does an AWS Organizations Service Control Policy (SCP) permit the request?
Does a Resource-based Policy (e.g., S3 Bucket Policy, KMS Key Policy) allow the action?
Does an Identity-based Policy (IAM Policy attached to the principal) allow the action?
Does the request fall within the ceiling defined by Permissions Boundaries and Session Policies?
Outcome: If any evaluation point yields an Explicit Deny, or if no policy grants an explicit Allow, the request is dropped (Implicit Deny).

IAM Permissions Boundaries

Defines the maximum permission ceiling for an IAM entity.
Enables delegating policy creation to development teams without risking privilege escalation. Even if a developer creates a new role with AdministratorAccess, the effective permissions will never exceed the attached Permissions Boundary (e.g., S3FullAccessBoundary).

Attribute-Based Access Control (ABAC) & Session Tags

Grants dynamic access based on matching metadata tags rather than maintaining complex, static JSON policies per user.
A single generic policy allows access if the principal's department or project tag matches the target resource tag (aws:ResourceTag/Project == aws:PrincipalTag/Project).

Cross-Account Access & ExternalID (Confused Deputy Protection)

When delegating role access to third-party SaaS vendors (e.g., Datadog, Databricks), requiring an sts:ExternalId condition inside the Role Trust Policy is mandatory.
Prevents the Confused Deputy attack vector, where an adversary manipulates the third-party platform into assuming another customer's role inside your AWS account.

💻 Practical Notes & Configuration Snippets

Attribute-Based Access Control (ABAC) Policy (Tag-Match Enforcement)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Project": "${aws:PrincipalTag/Project}"
        }
      }
    }
  ]
}

Essential AWS CLI Commands for IAM & STS

# Check current caller identity and active ARN
aws sts get-caller-identity

# Assume an IAM Role manually (returns temporary credentials)
aws sts assume-role \
  --role-arn "arn:aws:iam::123456789012:role/DevOpsAdminRole" \
  --role-session-name "CLI-Session" \
  --external-id "UniqueSecretId123"

# List IAM policies attached to a specific group
aws iam list-attached-group-policies --group-name "Engineering-Team"

⚠️ Gotchas & Common Pitfalls

Eventual Consistency in IAM APIs: IAM is a globally distributed control plane. Changes to policies, roles, or group memberships take several seconds to propagate across all AWS edge endpoints worldwide. Automation scripts and CI/CD pipelines must incorporate exponential backoff or brief delay retries immediately following IAM resource provisioning.
Identity-Based vs. Resource-Based Policy Overlaps: An Identity-based policy is a credential held by the caller; a Resource-based policy (e.g., S3 Bucket Policy, SQS Policy) is an access control list on the target resource. Within the same AWS account, if a resource-based policy grants explicit access, the caller can execute the action even without an attached identity-based IAM policy.
Wildcard (*) Operator Abuse: Using Action: "" and Resource: "" violates the Principle of Least Privilege and represents the primary security vulnerability identified during compliance audits.

💡 Production Best Practices

Enforce IMDSv2 Globally: Prevent SSRF-driven IAM token leakage from compute nodes by enforcing IMDSv2 (HttpTokens=required, HttpPutResponseHopLimit=1) across all EC2 launch templates and running instances.
Automate IAM Access Analyzer: Enable IAM Access Analyzer across the organization to automatically flag external, cross-account, and public access to S3 buckets, KMS keys, and IAM roles.
Root Account Monitoring & Alarms: Deploy an Amazon EventBridge rule tied to SNS alerts to notify security teams immediately via Slack or PagerDuty whenever the Root user logs into the AWS Management Console or triggers an API call.
Service Control Policies (SCPs) as Hard Guardrails: Implement SCPs at the AWS Organizations root to enforce organizational compliance (e.g., restricting available AWS Regions or denying modification to security audit log buckets). SCP Deny statements cannot be bypassed by any user within a member account, including IAM identities with full AdministratorAccess.

Kubernetes Architecture Deep Dive: From Resource Limits to Custom Operators

Enes Guler — Wed, 12 Aug 2026 12:33:54 +0000

Resource Management (Requests & Limits)

Resource management prevents Kubernetes workloads from depleting node hardware or causing resource contention among containers.

Resource Requests

The absolute minimum CPU and Memory guaranteed for a Pod to start. The Kubernetes Scheduler uses requests to determine node placement. If a node cannot fulfill the requested resources, the Pod will not be scheduled on that node.

Architectural Takeaway: Omitting requests leads to poor scheduling decisions, resulting in unbalanced cluster distribution and potential node starvation.

Resource Limits

The maximum ceiling of CPU and Memory a Pod is allowed to consume.

Memory (Non-Compressible Resource): If a process (e.g., a memory-heavy Pandas pipeline) exceeds its memory limit by even 1 MB, the Linux Kernel terminates the container with an OOMKilled (Out Of Memory Killed) exit code. This acts as a critical safety circuit breaker to constrain the blast radius and protect co-located services.
CPU (Compressible Resource): Unlike memory, exceeding CPU limits does not terminate the pod. Instead, the Linux Completely Fair Scheduler (CFS) enforces CPU Throttling. This constrains CPU usage, keeping the application alive but causing severe latency spikes during heavy traffic.

Architectural Takeaway: Memory limits protect nodes from crashing due to leaks, while improperly tuned CPU limits risk performance degradation via throttling even when the host node has idle CPU capacity.

Advanced Architectural Concepts

Beyond basic resource allocation, managing production Kubernetes clusters requires an understanding of hardware overcommitment, Linux Kernel throttling mechanics, and implicit Quality of Service (QoS) eviction hierarchies.

+-----------------------------------------------------------------------------------+
|                            Node Hardware Capacity                                 |
|                                                                                   |
|  +---------------------------+  +---------------------------+  +---------------+  |
|  | Guaranteed Pod            |  | Burstable Pod             |  | BestEffort    |  |
|  | Requests == Limits        |  | Requests < Limits         |  | No Req/Limits |  |
|  | (Lowest OOM Kill Priority)|  | (Medium Eviction Risk)    |  | (First Killed)|  |
|  +---------------------------+  +---------------------------+  +---------------+  |
+-----------------------------------------------------------------------------------+

Resource Overcommit

Overcommit occurs when the sum of all container resource limits on a node exceeds the node's actual physical hardware capacity, while the sum of resource requests remains within capacity bounds.

The Economics: In cloud infrastructure, running compute nodes at 20% average utilization is an expensive waste. Overcommit allows engineering teams to pack more workloads onto fewer nodes by betting that not all pods will hit their maximum resource limits simultaneously.
The Overcommit Trade-off:
- CPU Overcommit: Safe and manageable. CPU is a compressible resource; if demand exceeds capacity, execution speed slows down across pods.
- Memory Overcommit: High risk. Memory is non-compressible. If multiple pods suddenly spike toward their memory limits concurrently, the node runs out of physical RAM and swap. The Linux Kernel triggers OOMKilled events to forcibly terminate containers and reclaim memory.
Architectural Takeaway: Calculate the Overcommit Ratio (Sum of Limits) / (Physical Capacity) carefully. Overcommit CPU aggressively to save money, but keep memory overcommit conservative to avoid cascading application crashes.

The Linux CFS Quota & CPU Throttling Trap

Setting CPU limits relies on the Linux Kernel Completely Fair Scheduler (CFS) using cgroup enforcement. The Kernel evaluates CPU usage in enforced time windows, typically every 100ms (the CFS Period).

How the Trap Works: If a pod with a CPU limit of 1 vCPU (1000m) executes a multi-threaded operation that consumes 100ms worth of CPU processing time within the first 20ms of a period, the Kernel locks out the pod's CPU access for the remaining 80ms of that window.

100ms CFS Period Window
┌────────────────────────┬─────────────────────────────────────────┐
│  Multi-Threaded Burst  │          CPU Throttled / Locked         │
│   (Consumes 100ms CPU) │           (Latency Spikes Hit)          │
└────────────────────────┴─────────────────────────────────────────┘
 0ms                    20ms                                     100ms

The Idle Node Paradox: A pod can experience severe CPU Throttling (causing 500ms+ latency spikes in HTTP services) even when the underlying host worker node shows 80% idle CPU capacity.
Architectural Takeaway: Many enterprise SRE teams disable CPU limits entirely (Limits: Unset) for latency-sensitive microservices, relying strictly on well-tuned CPU Requests paired with Horizontal Pod Autoscaler (HPA) to handle traffic spikes safely.

Quality of Service (QoS) Classes & Eviction Order

Kubernetes automatically assigns every pod a QoS Class based on how its container requests and limits are configured. When a worker node experiences memory pressure, the Linux Kernel and Kubernetes Kubelet use the QoS class to determine eviction priority via the oom_score_adj metric.

Node Resource Exhaustion (Eviction Sequence)
────────────────────────────────────────────────────────────────────────►
[BestEffort Pods]   ───►   [Burstable Pods]   ───►   [Guaranteed Pods]
 (Terminated First)         (Terminated Second)       (Protected / Last)

Guaranteed (Highest Priority / Protected)
- Condition: Every container in the pod must explicitly specify both CPU and Memory, and Requests == Limits for all resources.
- Behavior: Highly stable. Granted an oom_score_adj of -997. These pods are the absolute last to be evicted or terminated during node memory starvation. Ideal for databases, core stateful sets, and critical payment services.
Burstable (Medium Priority / Standard)
- Condition: At least one container specifies a request or limit, but Requests != Limits (or CPU has a limit while Memory does not).
- Behavior: Allowed to burst beyond baseline when capacity allows. Evicted after all BestEffort pods are killed if memory pressure persists. Ideal for web APIs, background workers, and standard web applications.
BestEffort (Lowest Priority / Disposable)
- Condition: No requests or limits are defined for any container in the pod.
- Behavior: Assigned an oom_score_adj of 1000. Gets access to unallocated node resources, but is the first target for termination during node memory pressure. Ideal for non-critical batch processing, dev/test pods, or temporary log collectors.

Security and Observability

The control mechanisms that keep the system resilient before everything crashes or when a breach occurs.

RBAC (Role-Based Access Control)

The identity card and permission firewall for your code. It strictly limits what a Pod (or a user) can execute within the system.

ServiceAccount: The identity assigned to a Pod.
Role/ClusterRole: The explicit list of allowed permissions.
RoleBinding/ClusterRoleBinding: The bridge that staples an identity to a set of permissions. Even if a vulnerability leaks into the application code, RBAC prevents lateral movement and protects the underlying infrastructure.

Monitoring & Alerting

Kubernetes knows whether an application is running, but it is blind to the question: “Is the business logic actually behaving correctly?” To solve this:

Prometheus: Continuously scrapes and collects every metric (CPU, API response times, queue lengths, etc.) from the system.
Grafana: Transforms raw time-series metrics into visual, customizable dashboards.
Alertmanager: Triggers notifications (Slack, PagerDuty, Email) when response latency spikes from 200ms to 4 seconds, or when OOMKilled errors start popping up.

Service Mesh (Istio/Linkerd)

An infrastructure highway that manages, encrypts, and observes inter-service communication (East-West Traffic).

Sidecar Proxy: A lightweight proxy (like Envoy or NGINX) injected alongside your main application container inside the same Pod. It intercepts all inbound and outbound traffic completely transparently to your application code.
mTLS (Mutual TLS): Automatically encrypts network traffic between microservices in transit. Even if an attacker intercepts the network traffic, the payload remains unreadable.
Traffic Splitting: Allows you to route a portion of live traffic (e.g., 10%) to a new application version (Canary deployment) without altering a single line of application code.

Pod Lifecycle and Probes

Kubernetes evaluates the true health and readiness of an application inside a Pod using three distinct types of Probes.

Startup Probe

Ensures Kubernetes remains patient while an application boots up (for instance, legacy Java applications that might take over a minute to initialize). No other probes run until the Startup Probe succeeds. If this probe fails, Kubernetes assumes the application is stuck during startup and immediately restarts the Pod.

Liveness Probe

Answers the question: “Is this Pod alive?” If the application enters a deadlock or infinite loop, Kubernetes detects the stall via the probe, terminates the unresponsive container, and restarts it according to the restart policy.

Readiness Probe

Answers the question: “This Pod is alive, but is it ready to accept incoming user requests?” For example, the application process might be running, but it’s still establishing connection to a database. If this probe fails, Kubernetes does NOT kill or restart the Pod; it simply removes the Pod’s IP from the Service endpoints to stop routing traffic to it. Once the connection is established and the probe passes, traffic automatically resumes.

Networking Architecture: Ingress vs. Service

The networking layer that defines how traffic flows within the cluster and how external users access internal workloads.

ClusterIP (Service)

An internal extension line that allows Pods to communicate with one another within the cluster. It is completely isolated from the outside world.

NodePort (Service)

Exposes a specific port on every Node directly to the public internet. It is insecure and inefficient, making it rarely suitable for production environments.

LoadBalancer (Service)

Provisions a dedicated, paid external Cloud Load Balancer from your cloud provider (e.g., AWS ALB/NLB). Provisioning a separate cloud load balancer for every single microservice drastically inflates your cloud bill.

Ingress

A single Smart Traffic Router (Reverse Proxy) positioned at the cluster gateway. It terminates traffic from a single external Load Balancer and intelligently routes incoming requests to internal services based on domain names or URL path rules (/api, /auth). It minimizes infrastructure costs while providing a centralized point for TLS termination and traffic management.

Smart Placement

It is the set of rules that decides which applications run on which physical/virtual servers.

Taints

A security barrier (or label) applied to a Node. It is a node’s way of saying: “I am an expensive, GPU-heavy server. Standard web APIs without the right **tolerance* should stay away from me.”*

Tolerations

A specification written inside a Pod’s configuration. It is a Pod’s way of saying: “Yes, that server has a GPU taint, but I have the tolerance for it. You can place me there.”

Node Affinity

A specific requirement whispered by a Pod to the Kubernetes Control Plane. It is a Pod’s way of saying: “Do not place me randomly; run me strictly (or preferably) on Memory-Optimized nodes.”

Hard Constraint (required...): “I MUST have this node type, or do NOT deploy me at all.” (Strict rule)
Soft Constraint (preferred...): “I WOULD LIKE this node type, but if it’s not available, just put me anywhere.” (Preference)

Scaling and Autonomy

A set of mechanisms used to balance performance and infrastructure costs as workload demand increases or decreases.

HPA (Horizontal Pod Autoscaler)

Monitors CPU and RAM usage of Pods. If a Pod is under heavy load (heats up), HPA increases the replica count; as load drops (cools down), it scales back down. It is inherently sluggish because waiting for CPU metrics to spike takes time.

KEDA (Kubernetes Event-Driven Autoscaling)

Grants Kubernetes the ability to listen to external event sources (such as Redpanda, Kafka, S3, RabbitMQ, etc.). If a queue in Redpanda is empty, KEDA can scale the Pod replicas down to zero (Scale-to-Zero) to completely eliminate compute costs. If 10,000 messages suddenly hit the queue, it immediately spins up 50 Pods within seconds without waiting for CPU usage to rise.

Data and Storage

The layer that overcomes the ephemeral (temporary) nature of containers to manage persistent data.

CSI (Container Storage Interface)

The standard interface Kubernetes uses to communicate with cloud providers regarding storage. When a Pod needs a 50 GB disk for a database or AI model weights, the CSI driver dynamically provisions that disk from the cloud and physically attaches it to the node hosting the Pod. If the Pod dies and reschedules onto another node, CSI detaches the disk and reattaches it to the new node.

PV (Persistent Volume)

The physical/actual disk allocated to the cluster by the infrastructure administrator. It is a cluster-level resource independent of Pods, representing a declaration like: “I have a 100 GB SSD-backed disk available in AWS.”

PVC (Persistent Volume Claim)

A request for storage made by a developer (or application) to Kubernetes. It is a Pod’s way of saying: “I urgently need a 20 GB high-speed read/write disk.”

How PV and PVC Interact

When a developer creates a PVC, Kubernetes acts like a matchmaker. It scans the pool of available PVs. If it finds a PV that meets the requested capacity and access modes, it binds that PVC to the PV. The Pod only knows the name of the PVC and doesn’t care about the underlying storage implementation.

SC (StorageClass)

The set of instructions given to the CSI driver. When a developer specifies storageClassName: gp3 inside a PVC, Kubernetes checks if a matching PV exists. If no pre-provisioned PV is available, it immediately triggers the CSI driver. The CSI driver then provisions a 20 GB volume on AWS, registers it in Kubernetes as a new PV, and binds it to the PVC. This entire automated workflow is called Dynamic Provisioning.

Teaching K8s a New Language (Extending Kubernetes)

The capability that transforms Kubernetes from a simple application runner into a customizable platform builder.

CRD (Custom Resource Definition)

The process of extending Kubernetes’ built-in dictionary (which natively includes objects like Pods, Services, and Deployments) with your own custom resources. You define and register brand-new objects—such as ModelDeployment or PostgresDatabase—directly into the Kubernetes API database (etcd).

Operator Pattern

Kubernetes natively recognizes custom resources like ModelDeployment, but it doesn’t know what actions to take when one is created. An Operator is a custom controller written in languages like Go or Python that hooks into the Kubernetes control loop. When someone submits a ModelDeployment YAML manifest, the Operator wakes up and says: “Got it! I need to pull the AI model weights from S3, schedule a GPU-enabled node, and expose a REST API.” It then automatically orchestrates all the underlying standard Kubernetes objects on your behalf.

DEV Community: Enes Guler

AWS & SRE Field Manual (Part 4): Amazon EBS Architecture, Volume Performance & Kubernetes State

1. TL;DR & Problem Statement

2. Core Architecture & Key Components

2.1. Volume Types & Performance Spectrum

2.2. EBS Multi-Attach (Clustered Storage)

2.3. EBS Snapshots & Data Durability

3. Deep Dive Engineering & Architectural Comparison

4. Advanced Integrations & Kubernetes Binding

Kubernetes volumeBindingMode: WaitForFirstConsumer

Fast Snapshot Restore (FSR) & Pre-Warming (Lazy Loading)

5. Practical Notes & Configuration Snippets

Production Kubernetes StorageClass (gp3 with Delayed Binding)

Terraform: Provisioning an EBS gp3 Volume with Custom IOPS & Throughput

Volume Initialization / Pre-Warming via Linux CLI

6. Gotchas & Common Pitfalls

7. Production Best Practices

AWS & SRE Field Manual (Part 3): Amazon EC2 Architecture, Hardware Strategy & Modern Autoscaling

1. TL;DR & Problem Statement

2. Core Architecture & Key Components

2.1. Instance Families & Workload Mapping

2.2. Purchasing Models & Cost Engineering Strategies

2.3. Placement Groups (Physical Placement Strategies)

2.4. EBS-Optimized & Enhanced Networking (ENA)

3. Network Security: Security Groups vs. Network ACLs

4. Practical Notes & Configuration Snippets

Secure Metadata and Token Retrieval via IMDSv2

Karpenter NodePool Manifest (Modern Spot and On-Demand Node Autoscaling)

5. Gotchas & Common Pitfalls

6. Production Best Practices

AWS & SRE Field Manual (Part 2): Application (ALB) vs. Network (NLB) Load Balancing Architecture

1. TL;DR & Problem Statement

2. Core Architecture & Key Components

2.1. Application Load Balancer (ALB — OSI Layer 7)

2.2. Network Load Balancer (NLB — OSI Layer 4)

3. Deep Dive Engineering & Architectural Comparison

4. Advanced Integrations & Ingress Mechanics

AWS Load Balancer Controller (Instance Mode vs. IP Mode)

Preserving Real Client IP

Deregistration Delay (Connection Draining)

Cross-Zone Load Balancing

5. Practical Notes & Configuration Snippets

Kubernetes Ingress Manifest (ALB in IP Mode with SSL Redirect)

Terraform: Network Load Balancer with Cross-Zone Load Balancing

6. Gotchas & Common Pitfalls

7. Production Best Practices

AWS & SRE Field Manual (Part 1): Zero-Trust Identity & Access Management (IAM)

📌 TL;DR & Problem Statement

🏛️ Core Architecture & Key Components

1. IAM Users (Human Identities & Static Credentials)

2. IAM Groups (Role-Based Membership)

3. IAM Roles & AWS STS (Temporary Credentials & Workload Identity)

4. IAM Policies (Declarative Permission Documents)

⚙️ Deep Dive Engineering & Authorization Mechanisms

Policy Evaluation Logic (Order of Precedence)

IAM Permissions Boundaries

Attribute-Based Access Control (ABAC) & Session Tags

Cross-Account Access & ExternalID (Confused Deputy Protection)

💻 Practical Notes & Configuration Snippets

Attribute-Based Access Control (ABAC) Policy (Tag-Match Enforcement)

Essential AWS CLI Commands for IAM & STS

⚠️ Gotchas & Common Pitfalls

💡 Production Best Practices

Kubernetes Architecture Deep Dive: From Resource Limits to Custom Operators

Resource Management (Requests & Limits)

Resource Requests

Resource Limits

Advanced Architectural Concepts

Resource Overcommit

The Linux CFS Quota & CPU Throttling Trap

Quality of Service (QoS) Classes & Eviction Order

Security and Observability

RBAC (Role-Based Access Control)

Monitoring & Alerting

Service Mesh (Istio/Linkerd)

Pod Lifecycle and Probes

Startup Probe

Liveness Probe

Readiness Probe

Networking Architecture: Ingress vs. Service

Kubernetes `volumeBindingMode: WaitForFirstConsumer`

Production Kubernetes StorageClass (`gp3` with Delayed Binding)

Terraform: Provisioning an EBS `gp3` Volume with Custom IOPS & Throughput