DEV Community: Enes Guler

Terraform & IaC Field Manual (Part 1): Core Architecture, State Locking & Lifecycle Engineering

Enes Guler — Mon, 14 Sep 2026 06:45:34 +0000

Infrastructure as Code (IaC) is the practice of managing and provisioning computing infrastructure through machine-readable definition files, eliminating manual, error-prone console operations (Click-Ops). Mastering IaC requires a clear understanding of fundamental architectural contrasts and execution mechanics.

1. Core Architectural Paradigms

Imperative (Ansible / CLI)  ---> Defines the STEPS   ---> "How to build it"
Declarative (Terraform)     ---> Defines the TARGET  ---> "What to achieve"

Paradigm	Primary Tooling	Focus	State Tracking	Idempotency
Imperative	AWS CLI, Bash Scripts, Ansible (task mode)	How to build (Procedures)	Manual / External	Low (Script-dependent)
Declarative	Terraform, OpenTofu, CloudFormation	What to achieve (Desired State)	Managed Statefile (`.tfstate`)	High (Native Convergence)

1.1. Declarative vs. Imperative Paradigms

Imperative Paradigm (AWS CLI, Custom Scripts, Ansible):
- Execution Logic: Defines step-by-step procedural workflows (e.g., "Create an EC2 instance, attach a Security Group, verify storage").
- Trade-offs: Intermediate failures leave infrastructure in inconsistent, half-provisioned states. Re-runs struggle with idempotency.
Declarative Paradigm (Terraform, CloudFormation, OpenTofu):
- Execution Logic: Declares strictly the desired end-state of the target system (e.g., "Provision an instance with 2 vCPUs, 8GB RAM, attached to SG-X").
- Convergence & Directed Acyclic Graph (DAG): Terraform reconciles real-world state against declared code, calculates the execution diff, and builds a DAG to run independent API calls concurrently while serializing dependent resources.

1.2. Mutable vs. Immutable Infrastructure Strategy

Mutable Model:
[Base Instance] ---> [Manual SSH / Hotfix] ---> [Configuration Drift Accumulation]

Immutable Model (Terraform + Packer Pattern):
[Source Code] ---> [Packer Image Build] ---> [Terraform Deploy New VM] ---> [Destroy Old VM]

Mutable Infrastructure Model:
- Execution Logic: Servers are patched, updated, and reconfigured in-place over time via SSH or configuration management agents.
- Trade-offs: Leads to configuration drift, environment divergence, and complex, non-reproducible operational troubleshooting.
Immutable Infrastructure Model (Terraform + Packer Pattern):
- Execution Logic: Servers are never modified in-place. Updates trigger the provisioning of newly baked machine images alongside automated teardowns of legacy instances.
- Advantages: Absolute environment parity across staging and production, deterministic rollbacks, and complete elimination of runtime configuration drift.

1.3. Provisioning vs. Configuration Management Layering

Modern cloud-native operations enforce strict separation of operational boundaries:

Provisioning Layer (Terraform / OpenTofu): Orchestrates foundational cloud fabrics: VPCs, subnets, routing tables, security groups, IAM policies, managed databases (RDS), and Kubernetes control planes (EKS).
Configuration Management Layer (Ansible / Cloud-Init): In immutable workflows, configuration engines run strictly upstream inside image build pipelines (e.g., Packer) to generate static golden images, preventing ad-hoc mutation runs in production.

2. Core Architecture & Provider Decoupling

Terraform relies on a decoupled architecture split into two distinct tiers: Terraform Core and Providers, communicating over local Inter-Process Communication (IPC).

+--------------------------------------------------+
|                  Terraform Core                  |
|  (Parse HCL -> Build Graph -> Reconcile State)   |
+-------------------------+------------------------+
                          |
             RPC / gRPC Plugin Interface (IPC)
                          |
+-------------------------v------------------------+
|                   AWS Provider                   |
|       (Translates Core Request to Cloud API)     |
+-------------------------+------------------------+
                          |
                HTTPS / REST API Call
                          |
+-------------------------v------------------------+
|                  AWS Cloud API                   |
+--------------------------------------------------+

2.1. Terraform Core

Terraform Core is a statically compiled Go binary serving as the central orchestration brain:

HCL Parsing: Ingests, parses, and validates .tf and .tfvars configurations.
State Reconciliation: Compares live infrastructure captured inside .tfstate against declared declarations.
DAG Construction: Computes dependency trees across resources to optimize parallel execution paths.
Architectural Boundary: Core maintains zero native awareness of cloud APIs (AWS, Azure, GCP). It delegates all operational execution down to providers.

2.2. Providers (Plugin Ecosystem)

Providers are standalone binaries that bridge Terraform Core to upstream cloud APIs via gRPC/RPC:

API Translation: Translates abstract Core directives into platform-specific API payloads using vendor SDKs (e.g., AWS Go SDK).
Schema Definition: Exposes resource properties, constraints, and lifecycle handlers.
CRUD Operations: Implements Create, Read, Update, and Delete lifecycle execution logic.
Independent Versioning: Providers are maintained and released independently on the Terraform Registry.

2.3. Provider Resolution, Locking, and Optimization

1. Provider Source & Version Constraints

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

= 5.10.0: Strict version pinning.
>= 5.0, < 6.0: Restricts execution to the 5.x release series.
~> 5.0 (Pessimistic operator): Permits rightmost increments (5.1, 5.2) while locking against major breaking jumps (6.0).

2. Dependency Lock File (`.terraform.lock.hcl`)

Cryptographic Checksums: Stores h1: and zh: hashes across target platforms (Linux, macOS, Windows).
Supply-Chain Integrity: Guarantees deterministic provider downloads and blocks third-party binary tampering. Must be tracked in Git.

3. Plugin Caching for CI/CD Pipelines

Avoid redundant provider downloads across isolated runner jobs by setting a centralized cache path:

export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"

4. Multiple Provider Configurations (Provider Aliases)

# Default Provider (Primary Region)
provider "aws" {
  region = "eu-central-1"
}

# Aliased Provider (Disaster Recovery Region)
provider "aws" {
  alias  = "dr"
  region = "eu-west-1"
}

# Explicit Resource-to-Provider Binding
resource "aws_s3_bucket" "dr_backup" {
  provider = aws.dr
  bucket   = "app-disaster-recovery-backup-bucket"
}

3. State Management: The Source of Truth

The terraform.tfstate JSON file maps declared HCL resource blocks to actual real-world cloud identifiers (e.g., binding aws_instance.web to instance ID i-0a1b2c3d4e5f).

Local CLI / CI/CD (terraform apply)
              │
              ▼
┌──────────────────────────────┐
│  Acquire Lock via DynamoDB   │  ──► Blocks concurrent runs
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Pull State File from S3     │  ──► Decrypts in memory
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│ Compute Diff & Apply Changes │  ──► Executes Cloud API Calls
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Update & Push State to S3   │  ──► Saves new desired state
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  Release Lock via DynamoDB   │  ──► Unlocks for other engineers
└──────────────────────────────┘

3.1. Critical Risks Associated with State Files

Plaintext Sensitive Data Exposure: Resource attributes—including database credentials, private keys, and tokens—are persisted unencrypted inside .tfstate, regardless of sensitive = true tags.
Version Control Exclusion: State files must never be committed to Git repositories. Configure .gitignore accordingly:

# Terraform State & Local Runtime Artifacts
*.tfstate
*.tfstate.*
*.tfstate.backup
.terraform/
.terraform.lock.hcl.backup

Race Conditions: Simultaneous runs against an unlocked state file cause race conditions, dropped resources, and severe metadata corruption.

3.2. Remote Backends & State Locking

In production, state must reside centrally in Amazon S3 backed by distributed locking in Amazon DynamoDB:

terraform {
  backend "s3" {
    bucket         = "production-terraform-state-bucket"
    key            = "infrastructure/prod/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
  }
}

Storage & Encryption (S3): Enforces server-side encryption (AES-256 or aws:kms) and Object Versioning to permit state rollback during corruption incidents.
Lock Management (DynamoDB): Uses a primary partition key named LockID (String). Terraform writes a unique UUID on plan/apply, returning HTTP 423 Lock Collision errors to concurrent executions.

3.3. Production S3 Backend Hardening Checklist

Security Control	Configuration	Operational Purpose
Object Versioning	`versioning { enabled = true }`	Recovers state from accidental deletion or corruption.
Encryption-at-Rest	AWS KMS (`SSE-KMS`) / AES-256	Encrypts plaintext secrets present within the state JSON.
Public Access Block	`BlockPublicAcls = true`, `BlockPublicPolicy = true`	Eliminates accidental public internet exposure.
Enforce In-Transit TLS	S3 Bucket Policy (`aws:SecureTransport`)	Drops unencrypted HTTP connections from developer machines.

3.4. State CLI Operations & Emergency Management

Never edit .tfstate files directly in a text editor. Use the dedicated CLI subcommands:

# List tracked resources
terraform state list

# Inspect attributes of a tracked resource
terraform state show aws_instance.web

# Refactor resource address without triggering recreation in cloud
terraform state mv aws_instance.web aws_instance.api_gateway

# Stop tracking a resource without terminating the live cloud asset
terraform state rm aws_s3_bucket.legacy_logs

# Force release an orphaned lock caused by a crashed pipeline job
terraform force-unlock <LOCK-ID>

3.5. State Isolation: Directory-Based vs. Workspace Isolation

Directory-Based Isolation (Recommended for Environments): Separate directory structures for environments (dev/, staging/, prod/) and layers (networking/, compute/, databases/). Provides distinct remote state files, independent blast radiuses, and least-privilege IAM controls.
Workspace Isolation (terraform workspace): Employs a single configuration with dynamically prefixed state paths. Suitable for rapid feature-branch sandboxing, but anti-pattern for strict Prod/Non-Prod segregation due to shared state storage and identical IAM permissions.

4. Terraform Execution Lifecycle Commands

+-------------------+
|  terraform init   |  ---> Download Provider Plugins & Initialize Backend
+---------+---------+
          |
+---------v---------+
|  terraform plan   |  ---> Refresh State & Compute Proposed Execution Diff
+---------+---------+
          |
+---------v---------+
|  terraform apply  |  ---> Execute Cloud API Calls & Mutate State File
+---------+---------+
          |
+---------v---------+
| terraform destroy |  ---> Teardown Resources in Reverse Dependency Order
+-------------------+

4.1. Core Command Breakdown

terraform init: Prepares local environment, creates .terraform/, downloads provider plugins, configures remote backend connectivity, and checks out remote modules.
terraform fmt & terraform validate: Offline static code quality checks. Formats HCL canonical structure and validates schema rules without network overhead.
terraform plan: Queries live infrastructure to refresh state, calculates delta diffs, maps the execution graph, and reports planned changes (+, ~, -) in strict read-only mode.
terraform apply: Traverses the DAG, runs concurrent cloud API operations (default concurrency: parallelism=10), and writes state modifications atomically.
terraform destroy: Executes reverse DAG traversals to safely tear down infrastructure in strict inverted dependency sequence.

4.2. Production CI/CD Pattern: Deterministic Plans

[PR / Merge Request]  ---> terraform plan -out=tfplan  ---> Store Plan Artifact
                                                                   │
                                                                   ▼
[Approval Gate]       ---> terraform apply tfplan      ---> Guaranteed Execution Match

Generating binary plan artifacts eliminates the risk of applying out-of-band drifts between merge request reviews and deployment execution:

# Step 1: Generate a deterministic binary plan artifact
terraform plan -out=tfplan

# Step 2: Apply the evaluated artifact directly (skips confirmation prompts)
terraform apply tfplan

4.3. Targeted & Emergency Execution Flags

Command Flag	Use Case	Operational Impact
`-target=resource.name`	Isolated debugging or cycle breakage	Bypasses DAG validation; strictly restricted to incident recovery.
`-replace=resource.name`	Replaces deprecated `terraform taint`	Explicitly forces recreation of a single degraded resource.
`-refresh-only`	State drift alignment	Synchronizes statefile with real-world state without applying code mutations.
`-parallelism=N`	Concurrency & API rate limiting	Overrides maximum concurrent worker operations (default is 10).

5. Dependency Resolution: Implicit vs. Explicit Dependencies

Terraform automatically builds an internal Directed Acyclic Graph (DAG) to determine optimal execution sequences and parallelization paths.

IMPLICIT DEPENDENCY (Automatic)
[aws_security_group.api_sg] ──(exposes .id)──► [aws_instance.web_api]
                                               (Terraform waits automatically)

EXPLICIT DEPENDENCY (Manual Override via depends_on)
[aws_iam_role_policy_attachment] ──(depends_on)──► [aws_eks_identity_provider_config]
                                                   (Terraform forced to wait)

5.1. Implicit Dependencies (Attribute References)

Terraform detects natural resource dependencies by analyzing dynamic parameter references:

resource "aws_security_group" "api_sg" {
  name        = "api-security-group"
  description = "Allow inbound API traffic"
  vpc_id      = "vpc-123456"
}

resource "aws_instance" "web_api" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "c5.xlarge"

  # Implicit Dependency: Referencing the security group ID automatically 
  # sequences resource creation order in the execution graph.
  vpc_security_group_ids = [aws_security_group.api_sg.id]
}

5.2. Explicit Dependencies (`depends_on` Meta-Argument)

Required when an operational dependency exists without a direct attribute reference in HCL (e.g., IAM permission propagation prior to EKS cluster initialization):

resource "aws_eks_identity_provider_config" "example" {
  cluster_name = aws_eks_cluster.example.name

  # Explicit Dependency: Blocks initialization until IAM policy binding settles
  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy
  ]
}

5.3. Module-Level Dependencies

module "networking" {
  source = "./modules/vpc"
  cidr   = "10.0.0.0/16"
}

module "kubernetes_cluster" {
  source     = "./modules/eks"
  vpc_id     = module.networking.vpc_id
  subnet_ids = module.networking.private_subnets

  # Explicit Module Dependency: Enforces complete VPC routing prior to EKS deployment
  depends_on = [
    module.networking
  ]
}

5.4. Dependency Management Best Practices

Category	Best Practice	Anti-Pattern
Implicit vs. Explicit	Prefer natural attribute references (`resource.id`).	Blindly adding `depends_on` across configurations.
Concurrency Impact	Keep graph edges minimal to preserve `-parallelism`.	Over-constraining the DAG, forcing slow serial provisioning.
Circular Dependencies	Decouple resources to avoid cycle loops (`A -> B -> A`).	Declaring self-referencing inline rules within single blocks.

6. Resource Lifecycle Rules: Production Safety Mechanisms

DEFAULT LIFECYCLE (In-place Replacement)
[Destroy Old Resource] ──► (Downtime Window) ──► [Create New Resource]

CREATE_BEFORE_DESTROY LIFECYCLE (Zero-Downtime)
[Create New Resource] ──► [Health Verification] ──► [Destroy Old Resource]

6.1. Zero-Downtime Replacement (`create_before_destroy`)

By default, destructive resource updates cause Terraform to delete the legacy asset before provisioning the replacement. Setting create_before_destroy = true provisions the new instance first, verifies creation, rebinds dependencies, and terminates the obsolete asset.

6.2. Accidental Teardown Protection (`prevent_destroy`)

Serves as an engine-level guardrail for production databases, network gateways, and core storage buckets. If a plan indicates a destroy action for the resource, the execution engine errors out immediately.

6.3. Dynamic Drift Suppression (`ignore_changes`)

Prevents Terraform from reverting runtime changes made by external systems such as AWS Auto Scaling Groups (ASG), Kubernetes Horizontal Pod Autoscalers (HPA), or dynamic operational tagging.

6.4. Conditional Triggers & Custom Assertions

resource "aws_instance" "web_api" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "c5.xlarge"

  lifecycle {
    # Recreate instance if launch script changes
    replace_triggered_by = [
      aws_s3_object.bootstrap_script.version_id
    ]

    # Pre-execution validation check
    precondition {
      condition     = contains(["c5.xlarge", "c5.2xlarge"], var.instance_type)
      error_message = "Instance type must belong to the approved C5 tier."
    }

    # Post-execution state audit
    postcondition {
      condition     = self.root_block_device[0].encrypted == true
      error_message = "Root block device must be encrypted at rest."
    }
  }
}

6.5. Comprehensive Production Resource Template

resource "aws_instance" "web_api" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "c5.xlarge"

  tags = {
    Environment = "Production"
  }

  lifecycle {
    # Ensure zero downtime during instance replacements
    create_before_destroy = true

    # Prevent accidental destruction via Terraform CLI
    prevent_destroy = true

    # Ignore live updates made by external autoscalers or tagging engines
    ignore_changes = [
      tags,
      instance_type
    ]

    # Postcondition audit: Ensure instance is bound to the production VPC
    postcondition {
      condition     = self.vpc_security_group_ids != []
      error_message = "Production instances must be associated with at least one Security Group."
    }
  }
}

AWS & SRE Field Manual (Part 9): SRE Framework — SLI, SLO, SLA & Error Budget Engineering

Enes Guler — Sun, 13 Sep 2026 13:57:03 +0000

1. TL;DR & Problem Statement

Definition: A foundational Site Reliability Engineering (SRE) framework that defines, measures, tracks, and legally/operationally guarantees system reliability across three distinct abstraction layers.
Problem Solved: Eliminates ambiguity around the question "Is the system healthy and stable?" by aligning engineering velocity (shipping features fast) with product stability (keeping systems up) through shared, objective mathematical agreements.
Category: Site Reliability Engineering / Observability & Telemetry

2. Core Architecture & Key Components

SLI (Service Level Indicator)   ---> "What is the actual measurement right now?" (Prometheus Metric)
        │
        ▼
SLO (Service Level Objective)   ---> "What internal target must engineering hit?" (Team Goal / Error Budget)
        │
        ▼
SLA (Service Level Agreement)   ---> "What formal commitment do we make to customers?" (Legal / Financial Contract)

2.1. SLI (Service Level Indicator) - Empirical & Raw Telemetry

The actual, quantifiable metric representing real-time system performance and health emitted by telemetry platforms (Prometheus, CloudWatch, Datadog).

Availability / Success Rate: Ratio of successful requests to total requests (e.g., non-5xx responses vs. total requests).
Latency: Execution duration of incoming transactions (e.g., p95, p99 percentiles).
Throughput / Saturation: Ingested requests per second (RPS) and underlying resource exhaustion.
Example: "99.2% of HTTP requests returned 2xx/3xx over the last 5 minutes" or "Database p95 read latency is 120ms."

2.2. SLO (Service Level Objective) - Internal Engineering Target

The precise reliability target agreed upon internally by engineering, SRE, and product management.

The Golden Rule: The SLO must always be stricter than the SLA. This margin forms an operational safety buffer, allowing internal teams to remediate degradation before incurring external legal or financial penalties.
Error Budget: An SLO directly calculates an allowable margin of failure: $$\text{Error Budget} = 100\% - \text{SLO}$$ For a 99.9% monthly availability SLO, the maximum allowable downtime is approximately 43.8 minutes per 30-day window.
Example: "99.9% of API requests must complete successfully in under 200ms across any rolling 30-day window."

2.3. SLA (Service Level Agreement) - External Customer Contract

The legally binding agreement established between the service provider and paying end users.

Explicitly defines financial consequences, service credit refunds, and contractual termination remedies when breached. Audited directly by executive leadership and legal departments.
Example: "If monthly availability falls below 99.0%, the customer receives a 20% service credit refund on their subsequent billing invoice."

3. Deep Dive Engineering & Operational Matrix

Layer	Primary Audience	Metric Origin / Source	Impact of a Breach
SLI	SRE & On-Call Engineers	Telemetry & APM (Prometheus, OpenTelemetry)	Real-time dashboard spikes or threshold alert triggers
SLO	Engineering & Product Teams	Aggregate historical SLI evaluation window	Feature releases frozen; team pivots to reliability sprint
SLA	Customers, Legal & Executives	Contractual business compliance audits	Service credit payouts, financial penalties, contract breach

4. Advanced SRE Mechanisms & Reliability Patterns

Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)

Static threshold alerting (e.g., "Alert if error rate > 1% over 5 minutes") creates false-positive alert fatigue during minor spikes and completely misses slow budget erosion.

Burn Rate: The consumption velocity of an error budget relative to its standard exhaustion timeframe:
- 1x Burn Rate: Consumes 100% of the budget in exactly 30 days (normal baseline rate).
- 14.4x Burn Rate: Consumes 2% of the entire 30-day error budget in just 1 hour (requires immediate incident triage).
Production Alerting Logic: Uses intersecting multi-window verification:
- Short Window (14.4x / 2-minute to 1-hour window): Pages on-call engineers immediately (PagerDuty/Opsgenie).
- Long Window (6x / 6-hour window): Automatically generates a non-paging ticket for standard working hours investigation.

Composite SLIs & User Journey Mapping

Monitoring single metrics (e.g., isolated CPU utilization or individual database query speed) fails to reflect true user pain. Combine discrete multi-step transactions (e.g., User Authentication → Add to Cart → Payment Confirmation) into weighted Composite SLIs verified by Synthetic Canaries and Real User Monitoring (RUM).

The Over-Reliability Paradox & Chaos Injection

When an underlying service consistently runs far above its target SLO (e.g., delivering 100% uptime against a 99.9% SLO target), downstream dependent systems build false assumptions of perfection and omit necessary retries, timeouts, and circuit breakers. Teams use controlled chaos experiments (Chaos Engineering / Chaos Mesh) to deliberately burn excess error budgets, exposing hidden architectural fragility.

5. Practical Notes & Configuration Snippets

PromQL: Calculating 5-Minute Rolling HTTP Availability SLI

(
  sum(rate(http_requests_total{status!~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

OpenSLO Specification Manifest (SLO as Code)

apiVersion: openslo/v1
kind: SLO
metadata:
  name: checkout-service-availability
spec:
  service: checkout-service
  description: "99.9% availability for checkout operations over a rolling 30-day window"
  budgetingMethod: Occurrences
  objectives:
    - target: 0.999
      timeWindow:
        - duration: 30d
          isRolling: true
  indicator:
    metadata:
      name: checkout-availability-sli
    spec:
      ratioMetric:
        good:
          metricSource:
            type: prometheus
            query: sum(rate(http_requests_total{service="checkout", status!~"5.."}[5m]))
        total:
          metricSource:
            type: prometheus
            query: sum(rate(http_requests_total{service="checkout"}[5m]))

6. Gotchas & Common Pitfalls

100% Reliability Is an Anti-Pattern: Aiming for 100% availability is economically unsustainable. Because user edge networks (cellular, home Wi-Fi) have availability bounds below 99.9%, spending exponential engineering and cloud infrastructure budgets chasing the final 0.01% yields zero perceived user value.
Measurement Vantage Point Fallacy: Capturing SLIs exclusively inside deep backend container pods blinds the observability stack to edge networking failures, TLS handshake timeouts, and Ingress routing misconfigurations. Measure SLIs at the outermost ingress load balancer or API Gateway level.

7. Production Best Practices

Codified Error Budget Policy: Document clear operational rules for budget depletion. When an error budget reaches 0%, standard feature deployments to production are automatically blocked by the CI/CD pipeline, pivoting team capacity entirely to technical debt remediation, stability fixes, and chaos testing.
Eliminate Non-Actionable Alerting: Ensure production alerts are strictly actionable. If an alert does not require an immediate, human-driven operational intervention, it must not wake up on-call engineers.
Adopt SLO as Code (Sloth / Pyrra): Keep SLO definitions checked into Git repositories using OpenSLO or Sloth. Use CI/CD pipelines to automatically generate Prometheus alert rules and Grafana dashboards directly from declarative YAML files.

AWS & SRE Field Manual (Part 8): Amazon CloudWatch Architecture, Telemetry Pipelines & Production Observability

Enes Guler — Fri, 11 Sep 2026 13:17:42 +0000

1. TL;DR & Problem Statement

Definition: A native, serverless observability, telemetry, and monitoring suite within the AWS ecosystem that consolidates logs aggregation, time-series metrics collection, automated threshold alarms, and performance dashboards into a unified control plane.
Problem Solved: Eliminates the operational overhead and single-points-of-failure associated with deploying and maintaining self-hosted monitoring stacks (e.g., standalone Prometheus for metrics, Loki/ELK for logs, Alertmanager for notifications) while providing native, out-of-the-box visibility into AWS managed services.
Category: Management & Governance / Observability & Telemetry

2. Core Architecture & Key Components

                     ┌───────────────────────────────────────────────┐
                     │           AWS Infrastructure Resources        │
                     │   (EC2 Instances / EKS Pods / RDS / ALB)      │
                     └───────────────────────┬───────────────────────┘
                                             │
                                   Telemetry Pipeline
                                             │
        ┌────────────────────────────────────┼────────────────────────────────────┐
        ▼                                    ▼                                    ▼
┌─────────────────┐                  ┌─────────────────┐                  ┌─────────────────┐
│ CloudWatch Logs │                  │CloudWatch Metric│                  │ Custom App Data │
│  (Raw Streams)  │                  │ (System Stats)  │                  │ (Business KPIs) │
└────────┬────────┘                  └────────┬────────┘                  └────────┬────────┘
         │                                    │                                    │
         ▼                                    ▼                                    ▼
┌─────────────────┐                  ┌─────────────────┐                  ┌─────────────────┐
│ Logs Insights   │                  │CloudWatch Alarms│                  │   Dashboards    │
│  (SQL Querying) │                  │(SNS / AutoScale)│                  │ (Visual Graphs) │
└─────────────────┘                  └─────────────────┘                  └─────────────────┘

2.1. CloudWatch Logs (Log Groups & Streams)

Ingests, processes, and stores raw application and system logs from EC2 (via CloudWatch Agent), Amazon EKS containers (via Fluent Bit / AWS Distro for OpenTelemetry), and native AWS audit/access trails (RDS logs, ALB access logs, Lambda execution logs).
Log Groups & Streams: Log events reside inside distinct streams grouped hierarchically by application or microservice boundaries.

2.2. CloudWatch Logs Insights (Interactive Log Analytics)

A purpose-built, high-speed distributed query engine that executes interactive, SQL-like queries to parse, aggregate, and troubleshoot petabyte-scale log streams without standing up Elasticsearch clusters.

2.3. CloudWatch Metrics (System vs. Custom Telemetry)

Hypervisor-Level Metrics: AWS services emit out-of-the-box infrastructure metrics at 1-minute to 5-minute resolutions (e.g., EC2 CPU utilization, network packet counters, EBS volume operations).
OS-Level Metrics: Memory utilization (RAM) and disk capacity are managed within the guest OS and require the unified CloudWatch Agent to be installed.
Custom Metrics: Workloads can publish proprietary business domain metrics (e.g., CheckoutLatency, OrdersProcessedPerMinute) using the AWS SDK, CLI, or Embedded Metric Format.

2.4. CloudWatch Alarms & Composite Alarms

Metric Alarms: Continuously evaluate metric data points across rolling statistical windows (p95, p99, average) against explicit thresholds or dynamic machine-learning anomaly detection bands.
Composite Alarms: Combine multiple discrete alarms using boolean operators (AND, OR, NOT) to eliminate false-positive alert storms (e.g., "Trigger PagerDuty ONLY if CPU > 85% AND 5xx Error Rate > 5%").
Action Handlers: Dispatch webhook events to Amazon SNS (routing alerts to Slack/PagerDuty), trigger EC2 Auto Scaling policies, or invoke AWS Systems Manager Automation runbooks.

3. Deep Dive Engineering & Advanced Patterns

Embedded Metric Format (EMF) & High Cardinality

Publishing custom metrics individually via the PutMetricData API incurs substantial costs at scale and introduces strict API throttling/rate-limiting risks.

Mechanism: Applications output structured JSON logs containing a special _aws directive directly to stdout. CloudWatch Logs ingests the log stream, parses the EMF payload asynchronously, and extracts custom metrics in the background with zero PutMetricData API costs.

Container Insights with AWS Distro for OpenTelemetry (ADOT)

Rather than deploying legacy vendor-locked agents, modern EKS clusters deploy the ADOT Collector (OpenTelemetry standard).
ADOT collects distributed traces, container resource telemetry, and application metrics using vendor-neutral protocols (OTLP), exporting directly to CloudWatch, Amazon Managed Prometheus (AMP), or Jaeger.

Metric Math (Derived Expressions)

Enables dynamic calculations across multiple raw metrics on dashboards and alarms without deploying intermediate data-processing microservices (e.g., calculating percentage ratios: (ErrorCount / TotalRequests) * 100).

4. Practical Notes & Configuration Snippets

CloudWatch Logs Insights: Aggregating API 5xx Errors by Endpoint

fields @timestamp, @message, status, path
| filter status >= 500
| stats count(*) as error_count by path
| sort error_count desc
| limit 20

Embedded Metric Format (EMF) Structured Log Output

{
  "_aws": {
    "Timestamp": 1718000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Production/PaymentService",
        "Dimensions": [["Environment", "Operation"]],
        "Metrics": [
          {
            "Name": "ProcessingLatencyMs",
            "Unit": "Milliseconds"
          },
          {
            "Name": "SuccessfulTransactions",
            "Unit": "Count"
          }
        ]
      }
    ]
  },
  "Environment": "Production",
  "Operation": "StripeCheckout",
  "ProcessingLatencyMs": 84.5,
  "SuccessfulTransactions": 1,
  "RequestId": "req-9923847-acdf"
}

Terraform: Composite Alarm Definition

resource "aws_cloudwatch_composite_alarm" "high_severity_incident" {
  alarm_name        = "HighSeverity-PaymentService-Outage"
  alarm_description = "Fires only when high latency correlates with elevated 5xx error rate"

  alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.high_latency.alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.elevated_5xx_errors.alarm_name})"

  alarm_actions = [aws_sns_topic.pagerduty_alerts.arn]
  ok_actions    = [aws_sns_topic.pagerduty_alerts.arn]
}

5. Gotchas & Common Pitfalls

Default Log Retention (Never Expire): By default, CloudWatch Log Groups retain ingested logs indefinitely with no expiration date. Over months, unmanaged debug and access log groups accumulate silent, compounding storage costs. Always enforce an explicit retention policy (14–90 days) via Infrastructure-as-Code.
PutMetricData API Cost & Throttling: Sending thousands of high-frequency custom metrics synchronously via PutMetricData can generate unexpectedly large AWS bills and trigger API rate-limiting errors under peak traffic. Always buffer metrics or migrate to Embedded Metric Format (EMF).
Missing Standard RAM/Disk Metrics on EC2: Basic CloudWatch monitoring does not track guest operating system memory allocation or disk fill rates. Relying on default EC2 metrics for database or caching hosts will leave out-of-memory (OOM) conditions undetected.

6. Production Best Practices

Subscription Filters & Cross-Account Streaming: Stream high-throughput log groups in real-time to Amazon Kinesis Data Firehose or AWS Lambda to forward logs into cold S3 Parquet data lakes or external SIEM platforms (e.g., Splunk, Datadog), minimizing expensive long-term CloudWatch storage fees.
Log Anomaly Detection: Enable automated CloudWatch Log Anomaly Detection on critical log groups. The service uses machine learning to establish behavioral baselines and proactively flags unexpected error spikes or new unknown log formats without requiring static regex rules.
High-Resolution Metrics (1-Second Interval): For latency-critical financial or real-time trading services, configure custom metrics with StorageResolution: 1 to publish sub-minute (1-second) metrics for immediate incident triage.
CloudWatch Synthetics (Canaries): Deploy synthetic canary scripts (Node.js/Python headless Chromium) to continuously execute end-to-end user journeys (e.g., login, checkout flow) from global edge locations, verifying system availability even when real traffic is low.

AWS & SRE Field Manual (Part 7): Amazon RDS Deep Dive: Architecture and High Availability

Enes Guler — Sat, 05 Sep 2026 14:17:56 +0000

1. TL;DR & Problem Statement

Definition: A fully managed database service providing automated provisioning, high availability, patching, backups, and horizontal read scalability for relational engines (PostgreSQL, MySQL, MariaDB, Oracle, SQL Server, and Amazon Aurora).
Problem Solved: Eliminates the operational complexity, data-loss risk, and administrative overhead of running stateful database clusters inside self-managed containers or bare EC2 instances (e.g., storage corruption, manual failover scripts, snapshot replication, and split-brain scenarios).
Category: Database / Managed Storage

2. Core Architecture & Key Components

                     ┌──────────────────────────────────────────────────┐
                     │             AWS RDS Architecture                 │
                     └────────────────────────┬─────────────────────────┘
                                              │
                           ┌──────────────────┴──────────────────┐
                           │                                     │
                           ▼                                     ▼
         ┌───────────────────────────────────┐         ┌───────────────────┐
         │    Multi-AZ High Availability     │         │   Read Replicas   │
         │  (Zero Data Loss / Auto-Failover) │         │ (Scale-Out Reads) │
         └─────────────────┬─────────────────┘         └─────────┬─────────┘
                           │                                     │
      ┌────────────────────┴────────────────────┐                │
      ▼                                         ▼                ▼
┌──────────────────────────┐  Sync Write   ┌──────────────────────────┐  Async Rep  ┌──────────────────────────┐
│  Primary DB (AZ-1)       │ ────────────► │  Standby DB (AZ-2)       │ ───────────►│  Read Replica (AZ-3)     │
│  Reads & Writes          │               │  Passive / Auto-Failover │             │  Read-Only Queries       │
└──────────────────────────┘               └──────────────────────────┘             └──────────────────────────┘

2.1. Multi-AZ High Availability (Synchronous DR)

Mechanism: Provisions an active primary node in AZ-1 and a warm standby instance in AZ-2. Storage mutations (writes) are committed synchronously at the block level across zones before acknowledging the client transaction, guaranteeing RPO = 0 (Zero Data Loss).
Automated Failover: If the primary instance fails, RDS switches the database DNS CNAME record to the standby instance within 60–120 seconds. The connection string endpoint remains unchanged, requiring zero application reconfiguration.

2.2. Horizontal Read Scaling (Read Replicas)

Mechanism: Asynchronously streams database transaction logs (e.g., Write-Ahead Logs / WAL) to up to 15 read-only instances across Availability Zones or Regions.
Workload Isolation: Write operations are routed strictly to the Primary endpoint, while read-heavy workloads, analytics, and reporting dashboards query the Read Replica endpoints.

2.3. Amazon Aurora Architecture (Decoupled Compute & Storage)

Decoupling: Unlike standard RDS which uses attached Amazon EBS volumes, Aurora separates compute nodes from a shared, distributed, log-structured storage layer.
Quorum Model (4/6 Writes): Replicates 6 copies of data across 3 Availability Zones. Writes succeed as soon as 4 out of 6 storage nodes acknowledge the change, eliminating disk write serialization bottlenecks by pushing only Redo Logs over the network.

2.4. Amazon RDS Proxy (Managed Connection Pooler)

Sits between client applications (Kubernetes Pods, AWS Lambda) and the database engine.
Solves database resource exhaustion (max_connections limit) by multiplexing and pooling thousands of ephemeral client connections into a controlled set of persistent database sessions.
Reduces Multi-AZ failover time by up to 66% while preserving active client connections.

3. Deep Dive Engineering & Architecture Patterns

Replication Lag & Read-After-Write Consistency

Because Read Replicas sync asynchronously, a microsecond-to-second replication lag exists.

Pattern: To prevent users from seeing stale data immediately after an update (e.g., updating a user profile), write-heavy or latency-critical reads must bypass Read Replicas and target the Primary instance endpoint directly for a brief cooldown window.

Automated Backups & Point-in-Time Recovery (PITR)

RDS writes automated daily full-volume snapshots combined with continuous transaction log (WAL) ingestion to Amazon S3.
Enables restoration of the database to any millisecond within the backup retention window (1–35 days).

4. Practical Notes & Configuration Snippets

Terraform: PostgreSQL RDS Instance with Multi-AZ & Storage Autoscaling

resource "aws_db_instance" "production_db" {
  identifier                  = "prod-postgres-db"
  allocated_storage           = 100
  max_allocated_storage       = 1000 # Enables Storage Autoscaling up to 1TB
  engine                      = "postgres"
  engine_version              = "16.1"
  instance_class              = "db.r7g.xlarge" # AWS Graviton (ARM64)
  storage_type                = "gp3"
  multi_az                    = true
  publicly_accessible         = false
  auto_minor_version_upgrade  = true
  storage_encrypted           = true
  kms_key_id                  = aws_kms_key.db_encryption_key.arn
  db_subnet_group_name        = aws_db_subnet_group.db_private_subnets.name
  vpc_security_group_ids      = [aws_security_group.db_ingress.id]
  backup_retention_period     = 14
  deletion_protection         = true
  skip_final_snapshot         = false
  final_snapshot_identifier   = "prod-postgres-db-final-snapshot"
}

Database Failover Drill (AWS CLI)

# Manually trigger a Multi-AZ failover to test application resiliency
aws rds reboot-db-instance \
  --db-instance-identifier prod-postgres-db \
  --force-failover

5. Gotchas & Common Pitfalls

Database Subnet Group Placement: An RDS DB Subnet Group requires subnets in at least two distinct Availability Zones within the selected VPC. In production environments, these must always be Private Subnets with no Internet Gateways attached.
Storage Allocation Limits (Auto-Expand Cooldown): While RDS Storage Autoscaling automatically increases disk size when capacity drops below 10%, storage expansion enforces a mandatory 6-hour cooldown between expansion events. Rapid data ingestion during this window can lead to disk exhaustion (DiskFull).
Parameter Group Restarts: Modifying static engine parameters (e.g., shared_buffers, max_connections) requires a manual database reboot to take effect, whereas dynamic parameters apply immediately without instance restarts.

6. Production Best Practices

Graviton Processor Transition (db.m7g / db.r7g): Provision database instances on ARM64-based Graviton architectures to achieve up to 20–35% price-performance improvements over legacy x86 hardware.
Enable Performance Insights & Enhanced Monitoring: Standard CloudWatch metrics sample at 60-second intervals and hide OS-level context. Enhanced Monitoring provides 1-second OS process visibility, while Performance Insights breaks down queries by Average Active Sessions (AAS) and database wait states (e.g., db:lock, IO:DataFileRead).
Deletion Protection & Final Snapshots: Always set deletion_protection = true in production Infrastructure-as-Code to prevent accidental teardowns via Terraform or AWS Console misconfigurations.
Automate Certificate Rotation: RDS root SSL/TLS certificates expire periodically (e.g., every 5 years). Configure RDS automatic certificate rotation or integrate AWS EventBridge notifications to prevent catastrophic TLS handshake failures when certificate authorities expire.

AWS & SRE Field Manual (Part 6): Amazon S3 Architecture, Security Boundaries & Lifecycle Engineering

Enes Guler — Thu, 03 Sep 2026 07:45:15 +0000

1. TL;DR & Problem Statement

Definition: An industry-standard, serverless object storage service offering virtually unlimited scalability, high availability, advanced security guardrails, and 99.999999999% (11 9's) data durability. It stores data as discrete objects containing payload bytes, customizable metadata, and a globally unique identifier (GUID) rather than block storage (EBS) or directory file hierarchies (EFS).
Problem Solved: Eliminates disk capacity limits, physical storage maintenance, and high infrastructure costs associated with traditional file systems; makes petabyte-scale unstructured data (static web assets, application logs, backups, and analytical data lakes) addressable and retrievable directly over HTTPS.
Category: Storage / Object Storage

2. Core Architecture & Key Components

                             Client Request (HTTPS)
                                       │
                    ┌──────────────────┴──────────────────┐
                    ▼                                     ▼
        Public Internet (BPA Protected)         VPC Gateway Endpoint
                    │                               (Private Backbone)
                    └──────────────────┬──────────────────┘
                                       │
                                       ▼
                       ┌───────────────────────────────┐
                       │       Amazon S3 Bucket        │
                       │   (Global Unique Namespace)   │
                       └───────────────┬───────────────┘
                                       │
        ┌──────────────────────────────┼──────────────────────────────┐
        ▼                              ▼                              ▼
┌──────────────┐               ┌──────────────┐               ┌──────────────┐
│ S3 Standard  │ ──(Day 30)──► │ Standard-IA  │ ──(Day 90)──► │ Glacier Flex │
│ (Active Hot) │               │ (Cool Data)  │               │  (Deep Cold) │
└──────────────┘               └──────────────┘               └──────────────┘

2.1. Buckets & Global Namespace Architecture

Top-level logical containers for object storage.
While physically hosted in a specific AWS Region, bucket names share a globally unique namespace across all AWS accounts worldwide.
Addressing: Objects are deterministically addressed via ARNs or standardized HTTPS URLs:
- arn:aws:s3:::my-production-source-repo/app/config.json
- https://my-production-source-repo.s3.amazonaws.com/app/config.json

2.2. S3 Storage Classes & Cost-Performance Spectrum

S3 Standard: High-throughput, low-latency storage for actively accessed data (web assets, dynamic application files).
S3 Standard-IA (Infrequent Access): For data accessed less frequently but requiring millisecond retrieval times. Features lower base storage pricing but incurs per-GB data retrieval fees.
S3 Glacier Flexible / Deep Archive: Ultra-low-cost archival tiers for long-term compliance data. Retrieval latencies range from minutes to 12+ hours.
S3 Intelligent-Tiering: Automatically optimizes storage costs by continuously monitoring object access patterns and moving objects between frequent and infrequent tiers without performance impact or retrieval fees.

2.3. S3 Lifecycle Configuration Engine

Declarative automation rules that transition aging objects to cheaper storage classes (Transitions) or permanently purge them (Expiration).
Example Lifecycle Flow: Day 0: S3 Standard -> Day 30: Standard-IA -> Day 90: Glacier Flexible -> Day 365: Permanent Expiration.

2.4. S3 Versioning & MFA Delete

Retains previous versions of an object with a unique Version ID when overwritten, rather than destroying the underlying data.
Acts as the primary defense against accidental deletion, malicious writes, and ransomware. MFA Delete can be enforced to prevent permanent version destruction or versioning state changes without hardware token approval.

2.5. S3 Object Lock (WORM — Write Once, Read Many)

Enforces compliance and regulatory retention mandates (SEC Rule 17a-4, HIPAA) by preventing objects from being deleted or overwritten for a fixed retention period, even by the AWS Root account (Compliance Mode).

2.6. Replication Patterns (CRR / SRR)

Cross-Region Replication (CRR): Automatically and asynchronously copies objects across distinct AWS Regions for multi-region disaster recovery (DR) and localized latency reduction.
Same-Region Replication (SRR): Synchronizes objects across accounts or buckets within the same AWS Region for centralized logging and test/production environment isolation.

2.7. Pre-Signed URLs

Cryptographically signed, time-limited URLs (e.g., valid for 15 minutes) that grant temporary read or write permissions to unauthenticated clients (e.g., allowing a mobile app to upload a profile avatar directly to S3) without consuming application server compute or network bandwidth.

3. Deep Dive Engineering & Critical Security Controls

S3 Block Public Access (BPA)

An account-level and bucket-level circuit breaker that unconditionally overrides permissive ACLs and bucket policies, preventing accidental exposure of sensitive buckets to the public internet.

VPC Gateway Endpoints

Routes all S3 traffic originating from EC2, ECS, or EKS workloads directly through AWS's internal private network backbone.
Eliminates NAT Gateway data processing fees, prevents egress traffic over the public internet, and enforces strict endpoint boundary policies.

Strong Read-After-Write Consistency

Amazon S3 automatically provides strong read-after-write consistency for PUT and DELETE requests of objects in all AWS Regions with zero eventual-consistency replication lag.

4. Practical Notes & Configuration Snippets

S3 Bucket Policy (Restricting Ingress to Specific IAM Role and VPC Endpoint)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictToVPCAndRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/EKS-App-Pod-Role"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-production-source-repo/*",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-0123456789abcdef0"
        }
      }
    }
  ]
}

Essential AWS CLI Commands for S3

# Sync local directory to S3 bucket efficiently (copies only changed files)
aws s3 sync ./build-artifacts s3://my-production-source-repo/builds/ --delete

# Generate a pre-signed URL for temporary object upload (valid for 15 minutes)
aws s3 presign s3://my-production-source-repo/uploads/user-avatar.png --expires-in 900

# Enable versioning on a bucket
aws s3api put-bucket-versioning \
  --bucket my-production-source-repo \
  --versioning-configuration Status=Enabled

5. Gotchas & Common Pitfalls

Small File Lifecycle Transition Costs: Transitioning objects smaller than 128 KB to Standard-IA or Glacier is uneconomical. S3 bills storage for a minimum of 128 KB per object, and the per-request transition API costs often exceed the storage savings.
Delete Markers in Versioned Buckets: Issuing a simple DELETE request against a versioned object does not physically delete the data; it simply places a zero-byte Delete Marker on top of the version stack. To reclaim storage capacity, you must delete the explicit Version ID or configure Lifecycle rules to purge expired object delete markers.
KMS API Throttling with SSE-KMS: Encrypting high-velocity buckets with standard AWS KMS keys can hit default account KMS request rate limits (throttling) under high RPS workloads. Enable S3 Bucket Keys to reduce KMS API calls by up to 99%.

6. Production Best Practices

Enforce In-Transit Encryption (aws:SecureTransport): Add an explicit Deny statement in your bucket policy for any request where "aws:SecureTransport": "false" to reject non-HTTPS connections unconditionally.
S3 Inventory over ListObjectsV2: For buckets holding millions of objects, running ListObjectsV2 API scans introduces extreme latency and high request fees. Use S3 Inventory to output daily or weekly CSV/Parquet metadata reports directly to S3.
Abort Incomplete Multipart Uploads: Large file uploads (greater than 100 MB) use multipart chunking. Always configure a bucket lifecycle rule to automatically abort and clean up failed, partial multipart uploads after 7 days to eliminate hidden orphan storage costs.

AWS & SRE Field Manual (Part 5): Amazon EKS Architecture, Pod Networking & Next-Gen Scaling

Enes Guler — Wed, 02 Sep 2026 08:35:10 +0000

1. TL;DR & Problem Statement

Definition: A managed Kubernetes service that offloads the operational complexity of deploying, scaling, and maintaining the Kubernetes Control Plane while integrating natively with core AWS networking, identity, and storage backbones.
Problem Solved: Eliminates single points of failure and maintenance overhead associated with self-managed etcd and master nodes (manual backup/restore, OS patching, control plane scaling) and bridges the gap between Kubernetes orchestration and cloud-native AWS primitives.
Category: Containers & Compute

2. Core Architecture & Key Components

                     ┌──────────────────────────────────────────────────┐
                     │      AWS Managed Control Plane (Multi-AZ)        │
                     │   kube-apiserver  |  etcd  | controller-manager  │
                     └────────────────────────┬─────────────────────────┘
                                              │
                                  Single Endpoint (HTTPS)
                                              │
       ┌──────────────────────────────────────┼──────────────────────────────────────┐
       ▼                                      ▼                                      ▼
┌──────────────┐                      ┌──────────────┐                      ┌──────────────┐
│ Self-Managed │                      │ Managed Node │                      │ AWS Fargate  │
│  EC2 Nodes   │                      │ Groups (MNG) │                      │ (Serverless) │
└──────────────┘                      └──────────────┘                      └──────────────┘

2.1. Managed Control Plane

AWS provisions and maintains kube-apiserver, etcd, kube-scheduler, and kube-controller-manager across at least 3 Availability Zones (AZs) behind redundant Network Load Balancers.
Features automated etcd snapshot backups, control plane autoscaling, and version patch rollouts with an enterprise-grade availability SLA.

2.2. Data Plane Abstraction Models

Self-Managed Nodes: Legacy pattern where instances are provisioned, bootstrapped, and maintained manually by the operator.
Managed Node Groups (MNG): AWS automates EC2 provisioning, AMI upgrades, and graceful node draining (kubectl drain) during cluster lifecycle operations while retaining EC2 instance visibility.
AWS Fargate: Serverless compute engine that removes the node abstraction entirely. Each pod runs inside its own isolated VM boundary with right-sized CPU/memory allocation and zero OS-level server maintenance.

2.3. Next-Gen Node Autoscaling (Karpenter vs. Cluster Autoscaler)

Traditional Cluster Autoscaler (CA) depends on fixed Auto Scaling Groups (ASGs), causing slow scale-up times and instance over-provisioning.
Karpenter: A declarative, group-less node autoscaler. It monitors pending unschedulable pods, calculates exact aggregate compute/memory requirements and topology spread constraints, and calls the AWS EC2 fleet API directly to provision right-sized Spot or On-Demand instances within seconds.

3. Deep Dive Integrations & Advanced Mechanisms

A. Networking: AWS VPC CNI & Prefix Delegation

Standard Kubernetes overlays (e.g., Flannel) wrap packets inside VXLAN and apply SNAT. The AWS VPC CNI plugin assigns real, routable VPC subnet IPs directly to each Pod ENI.
Prefix Delegation (ENABLE_PREFIX_DELEGATION=true): Standard EC2 instances have strict ENI/IP limits, restricting pod density per node. Prefix delegation assigns /28 IPv4 subnets (16 IPs per slot) instead of single IPs, multiplying maximum pod density per node without requiring larger instance types.

B. Workload Identity: IRSA vs. EKS Pod Identities

IAM Roles for Service Accounts (IRSA): Uses OIDC federation. A Kubernetes ServiceAccount is annotated with an IAM Role ARN, and AWS STS injects short-lived credentials via projected volume tokens.
EKS Pod Identities: The modern, streamlined alternative to IRSA. Eliminates manual OIDC provider setup and complex trust relationship JSONs. Permissions are mapped directly via the EKS API and enforced by the local EKS Pod Identity agent daemonset.

C. Storage: EBS & EFS CSI Drivers

Bridges Kubernetes dynamic volume provisioning (PersistentVolumeClaim) with native AWS storage classes (gp3, io2, EFS).
Handles dynamic volume creation, attachment across AZs, and automated disk resizing through standard Kubernetes manifests.

D. Endpoint Access Control (Private Clusters)

Public and Private: API server is accessible from the internet; worker node traffic remains within the private VPC.
Private Only (Production Standard): Disables public internet access to kube-apiserver completely. Cluster management requires direct VPC peering, AWS Client VPN, or an internal Bastion host.

4. Practical Notes & Configuration Snippets

EKS Pod Identity Association (Terraform)

resource "aws_eks_pod_identity_association" "s3_reader" {
  cluster_name    = "production-cluster"
  namespace       = "default"
  service_account = "s3-reader-sa"
  role_arn        = aws_iam_role.s3_reader_role.arn
}

Enabling Prefix Delegation on AWS VPC CNI

# Enable IPv4 prefix delegation for increased pod density
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

# Set warm prefix target to optimize IP allocation latency
kubectl set env daemonset aws-node -n kube-system WARM_PREFIX_TARGET=1

Verify Cluster Endpoint and Worker Nodes

# Update local kubeconfig to point to private/public EKS cluster
aws eks update-kubeconfig --region us-east-1 --name production-cluster

# Check all node statuses and their instance types provisioned by Karpenter
kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/capacity-type

5. Gotchas & Common Pitfalls

VPC Subnet IP Exhaustion: Because the VPC CNI assigns native subnet IPs to every single pod, running large clusters on narrow VPC CIDRs (e.g., /20 or /24) will quickly deplete available subnet IPs, preventing new EC2 instances and pods from launching. Always design secondary CIDR blocks (e.g., 100.64.0.0/16 CGNAT) for pod networking.
EBS Cross-AZ Mounting Failure: Amazon EBS volumes are strictly zonal. A pod bound to an EBS volume in us-east-1a cannot mount that volume if rescheduled by Kubernetes onto a node in us-east-1b. Use EBS only with single-AZ stateful workloads or migrate to Amazon EFS / Amazon FSx for multi-AZ shared storage.
CoreDNS Throttling: Rapid pod scale-ups can overwhelm the default 2-replica CoreDNS deployment, causing intermittent DNS query timeouts (5-second lookups). Deploy NodeLocal DNSCache and configure horizontal autoscaling for CoreDNS.

6. Production Best Practices

Control Plane Logging to CloudWatch: Always enable all 5 control plane log types in production (api, audit, authenticator, controllerManager, scheduler) to maintain compliance and facilitate root cause analysis during incidents.
AWS Load Balancer Controller over In-Tree Service Controller: Avoid using the deprecated built-in Kubernetes service load balancer. Use the AWS Load Balancer Controller to provision target-group-binding Application Load Balancers (ALBs) routing traffic directly to Pod IPs in IP-mode, bypassing kube-proxy NodePort latency.
Strict AWS Security Group per Pod: For regulated workloads (PCI-DSS / HIPAA), use Security Groups for Pods to enforce native AWS stateful firewall rules directly at the Kubernetes pod ENI level instead of relying solely on Kubernetes NetworkPolicies.
Karpenter Consolidation Policies: Enable consolidationPolicy: WhenUnderutilized in Karpenter NodePools to continuously repack workloads onto fewer or cheaper compute instances during low-traffic periods, reducing idle cluster costs by 30% to 50%.

AWS & SRE Field Manual (Part 4): Amazon EBS Architecture, Volume Performance & Kubernetes State

Enes Guler — Tue, 01 Sep 2026 08:54:53 +0000

1. TL;DR & Problem Statement

Definition: A high-performance, persistent block-level storage service designed for use with Amazon EC2 instances and Kubernetes worker nodes. Unlike ephemeral instance store volumes, data on an EBS volume persists independently of the lifecycle of the attached compute instance.
Problem Solved: Provides durable, stateful block storage for databases, file systems, and enterprise applications with independent provisioning of capacity, IOPS, and throughput, backed by automated point-in-time incremental snapshots.
Category: Storage / Persistent Block Storage

2. Core Architecture & Key Components

                     ┌──────────────────────────────────────────────┐
                     │           AWS EKS / EC2 Compute              │
                     └──────────────────────┬───────────────────────┘
                                            │
                               Storage Attachment (RWO / Multi-Attach)
                                            │
      ┌─────────────────────────────────────┴─────────────────────────────────────┐
      ▼                                                                           ▼
┌─────────────────────────────────┐                             ┌─────────────────────────────────┐
│     Amazon EBS Volume (gp3)     │                             │        EBS Snapshot Engine      │
│  Independent IOPS & Throughput  │                             │  Block-Level Incremental Backup │
└────────────────┬────────────────┘                             └────────────────┬────────────────┘
                 │                                                               │
                 ▼                                                               ▼
┌─────────────────────────────────┐                             ┌─────────────────────────────────┐
│ Dynamic Kubernetes PVC Binding  │                             │   Stored Durably in Amazon S3   │
└─────────────────────────────────┘                             └─────────────────────────────────┘

2.1. Volume Types & Performance Spectrum

General Purpose SSD (gp3): The modern cloud standard. Decouples storage volume capacity (GiB) from performance metrics (IOPS and throughput in MB/s), delivering a baseline of 3,000 IOPS and 125 MB/s free with every volume.
General Purpose SSD (gp2 - Legacy): Couples IOPS linearly to capacity (3 IOPS per GiB). Scaling performance requires over-provisioning unused disk size.
Provisioned IOPS SSD (io2 Block Express): Sub-millisecond latency SAN-grade storage delivering up to 256,000 IOPS, 4,000 MB/s throughput, and 99.999% durability for mission-critical database engines (Oracle, SAP HANA, Microsoft SQL Server).

2.2. EBS Multi-Attach (Clustered Storage)

Enables attaching a single Provisioned IOPS (io2/io1) volume concurrently to up to 16 nitro-based EC2 instances within the same Availability Zone.
Requires a cluster-aware file system (e.g., GFS2, OCFS2) to manage write locks and prevent data corruption.

2.3. EBS Snapshots & Data Durability

Incremental Block-Level Backups: Snapshots capture only the delta (modified blocks) since the previous snapshot, stored durably inside Amazon S3 across multiple Availability Zones.
Crash-Consistent vs. Application-Consistent: Snapshots taken on running instances are crash-consistent; freeze I/O or flush database buffers to disk prior to snapshot creation for application consistency.

3. Deep Dive Engineering & Architectural Comparison

Attribute	Legacy `gp2` Volume	Modern `gp3` Volume	Provisioned IOPS `io2 Block Express`
Baseline Performance	Tied to capacity (3 IOPS/GiB)	Fixed 3,000 IOPS & 125 MB/s	Configured per provisioned IOPS
Performance Scaling	Must expand disk capacity	Scale IOPS up to 16,000 independently	Scale up to 256,000 IOPS
Max Throughput	250 MB/s	1,000 MB/s	4,000 MB/s
Durability SLA	99.8%–99.9%	99.8%–99.9%	99.999%
Multi-Attach Support	No	No	Yes (Up to 16 EC2 nodes in same AZ)
Cost Profile	Expensive due to capacity bloat	Up to 20% cheaper per GiB than `gp2`	Premium pricing for extreme IOPS

4. Advanced Integrations & Kubernetes Binding

Kubernetes `volumeBindingMode: WaitForFirstConsumer`

Standard EBS volumes are strictly zonal (locked to a specific AZ such as us-east-1a).
If volumeBindingMode: Immediate is used, the EBS CSI driver provisions the volume in an arbitrary AZ upon PVC creation. If the pod is subsequently scheduled on a worker node in a different AZ, the pod fails to start with FailedAttachVolume.
WaitForFirstConsumer delays volume creation until the Kubernetes scheduler assigns the pod to a specific node, ensuring the EBS volume is dynamically provisioned in the exact matching AZ.

Fast Snapshot Restore (FSR) & Pre-Warming (Lazy Loading)

When restoring an EBS volume from an S3 snapshot, storage blocks are pulled on-demand (lazy loaded) upon first access, causing a temporary latency spike.
Mitigation: Enable Fast Snapshot Restore (FSR) on the snapshot for instantaneous maximum performance, or execute block-level sequential reads (fio or dd) to pre-warm the volume before routing production traffic.

5. Practical Notes & Configuration Snippets

Production Kubernetes StorageClass (`gp3` with Delayed Binding)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  iops: "4000"
  throughput: "250"
  encrypted: "true"

Terraform: Provisioning an EBS `gp3` Volume with Custom IOPS & Throughput

resource "aws_ebs_volume" "database_data" {
  availability_zone = "us-east-1a"
  size              = 200 # 200 GiB
  type              = "gp3"
  iops              = 5000
  throughput        = 250
  encrypted         = true
  kms_key_id        = aws_kms_key.ebs_encryption_key.arn

  tags = {
    Name        = "prod-postgres-data-vol"
    Environment = "production"
  }
}

Volume Initialization / Pre-Warming via Linux CLI

# Read all blocks sequentially from newly attached raw EBS block device
sudo dd if=/dev/xvdf of=/dev/null bs=1M status=progress

6. Gotchas & Common Pitfalls

Cross-AZ Attachment Impossibility: EBS volumes cannot attach across Availability Zone boundaries. Stateful workloads requiring multi-AZ concurrent file sharing must use Amazon EFS (NFS) or Amazon FSx rather than EBS.
Dynamic Volume Expansion Limits: While AWS allows online expansion of EBS volume size without downtime, reducing the size of an EBS volume is not supported by the AWS API. To shrink a disk, you must provision a smaller volume and copy filesystems over.
Volume Modification Rate Limits: AWS enforces a mandatory 6-hour cooldown period between modifications (size, IOPS, throughput) on a single EBS volume.

7. Production Best Practices

Always Migrate Legacy gp2 to gp3: Migrating volumes from gp2 to gp3 is a live, zero-downtime operation using the ModifyVolume API. It instantly yields a 20% baseline cost reduction while decoupling IOPS scaling.
Enforce Account-Level EBS Encryption by Default: Enable the account-level setting EnableEbsEncryptionByDefault across all AWS regions to guarantee that unencrypted block storage cannot be accidentally provisioned via CLI, Console, or CI/CD pipelines.
AWS Data Lifecycle Manager (DLM) for Automated Backups: Use Amazon Data Lifecycle Manager to automate snapshot creation schedules, cross-region replication for disaster recovery (DR), and automated retention cleanup policies without writing custom Lambda cron scripts.
Monitor VolumeQueueLength & VolumeThroughputPercentage: Use CloudWatch metrics to detect storage bottlenecks. A continuously elevated VolumeQueueLength indicates that application I/O requests are queuing due to exhausted IOPS limits.

AWS & SRE Field Manual (Part 3): Amazon EC2 Architecture, Hardware Strategy & Modern Autoscaling

Enes Guler — Mon, 31 Aug 2026 07:00:20 +0000

1. TL;DR & Problem Statement

Definition: A core cloud compute service providing resizable, secure, and on-demand virtual machines (instances) with customizable hardware architectures, CPU/memory ratios, and operating systems.
Problem Solved: Eliminates physical server procurement lead times (weeks to months), datacenter maintenance overhead, and capacity planning risks by enabling instant provisioning of scalable infrastructure for containerized workloads, databases, and application backends.
Category: Compute / Infrastructure-as-a-Service (IaaS)

2. Core Architecture & Key Components

                                  ┌───────────────────────────────────────────┐
                                  │            Amazon EC2 Security            │
                                  │           Stateful Firewalls              │
                                  └─────────────────────┬─────────────────────┘
                                                        │
                                                        ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                       Workload Optimization Spectrum                                        │
│                                                                                                             │
│  ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────────┐   ┌───────────────────┐  │
│  │ General Purpose (T/M) │   │ Compute Optimized (C) │   │ Memory Optimized (R)  │   │  GPU / AI (G/P)   │  │
│  │ Burstable / Balanced  │   │ High CPU / Analytics  │   │ In-Memory / Redis     │   │ Machine Learning  │  │
│  └───────────────────────┘   └───────────────────────┘   └───────────────────────┘   └───────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

2.1. Instance Families & Workload Mapping

General Purpose (T & M Series): Delivers balanced compute, memory, and networking resources. Ideal for web servers, microservices, and development environments.
- T-Series Burstable Capacity: Operates on a CPU Credit model. Instances accumulate credits during idle periods and burst to full CPU capacity when traffic spikes occur, providing a cost-effective compute base for baseline workloads.
Compute Optimized (C Series): Engineered for high-performance processors. Tailored for compute-intensive applications such as batch processing, media encoding, high-traffic web APIs, and scientific modeling.
Memory Optimized (R Series): Provides high RAM-to-vCPU ratios. Designed to handle memory-heavy workloads, including in-memory caches (Redis, Valkey, Memcached), large-scale data processing pipelines (Pandas/Spark), and enterprise relational databases.
Accelerated Computing / GPU (G & P Series): Equipped with high-performance hardware graphics processors (NVIDIA GPUs). Essential for deep learning model training, generative AI inference, high-end rendering, and complex parallel matrix calculations.
- Kubernetes Connection: In EKS clusters, G/P series instances are assigned Taints (e.g., gpu-workload=true:NoSchedule) to isolate them, ensuring only dedicated machine learning pods with matching Tolerations deploy onto expensive GPU nodes.

2.2. Purchasing Models & Cost Engineering Strategies

On-Demand Instances: Billed by the second with zero long-term commitments or upfront costs. This is the most expensive purchasing model, intended primarily for short-term testing, dynamic unpredictable spikes, and initial environment setups.
Savings Plans & Reserved Instances (RI): Committing to run a specific instance profile or consistent compute usage over a 1-year or 3-year term yields discounts ranging from 30% to 72% compared to On-Demand rates. Ideal for baseline production workloads and database clusters that run non-stop.
Spot Instances: Allows purchasing spare AWS compute capacity at steep discounts (up to 90% off On-Demand pricing).
- The Interruption Risk: If AWS requires the capacity back for On-Demand customers, it issues a 2-minute warning notification (Spot Interruption Notice) before forcibly terminating the Spot instance.
- Kubernetes Node Strategy: Stateless web microservices running in Amazon EKS can be deployed onto Spot instance node groups. When AWS reclaims a Spot node, the EKS control plane uses the 2-minute notice window to gracefully drain node connections (kubectl drain) and reschedule pods onto remaining healthy nodes, dramatically lowering compute overhead.

2.3. Placement Groups (Physical Placement Strategies)

Cluster: Places instances physically close together inside a single Availability Zone to achieve ultra-low network latency and high packet-per-second throughput (ideal for HPC and distributed ML model training).
Spread: Strictly places each instance across distinct physical hardware racks with independent power and network paths to minimize correlated hardware failure risks for critical single-node instances.
Partition: Divides groups of instances into logical partitions across distinct hardware racks, ensuring that large distributed systems (Kafka, Cassandra, HDFS) do not share common rack failures.

2.4. EBS-Optimized & Enhanced Networking (ENA)

EBS-Optimized Instances: Allocates dedicated, isolated network bandwidth strictly for EBS block storage I/O, preventing storage traffic from competing with general application networking.
Enhanced Networking (ENA): Uses SR-IOV (Single Root I/O Virtualization) to bypass the hypervisor layer, minimizing CPU utilization and providing up to 100+ Gbps network bandwidth with ultra-low jitter.

3. Network Security: Security Groups vs. Network ACLs

Incoming Request (Port 8080)
                             │
                             ▼
              ┌─────────────────────────────┐
              │    EC2 Security Group       │
              │  (Implicit Default Deny)    │
              └──────────────┬──────────────┘
                             │  Matched Ingress Rule?
                             ├─── No  ──► [Dropped / Blocked]
                             │
                             └─── Yes ──► [Allowed to Instance]
                                              │
                                              ▼
                                 Auto-Allowed Return Traffic
                                   (Stateful Mechanism)

Security Groups (Virtual Firewall):
- Operates at the virtual network interface (ENI) and instance level.
- Implicit Default Deny: By default, all incoming traffic is blocked until explicit ingress rules are defined.
- Stateful Mechanism: Connection tracking is fully stateful. If an inbound request is permitted by an ingress rule, the corresponding outbound response traffic is automatically allowed out of the instance, regardless of outbound/egress rules.
Network ACLs (NACLs):
- Operates at the subnet boundary level.
- Stateless Mechanism: Rules are evaluated sequentially (by rule number) and are completely stateless, requiring explicit allow rules in both inbound and outbound directions (including ephemeral port ranges for return traffic).

4. Practical Notes & Configuration Snippets

Secure Metadata and Token Retrieval via IMDSv2

# Step 1: Request a short-lived session token (TTL: 21600 seconds)
TOKEN=$(curl -s -X PUT "[http://169.254.169.254/latest/api/token](http://169.254.169.254/latest/api/token)" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

# Step 2: Fetch instance role credentials securely using the token
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  [http://169.254.169.254/latest/meta-data/iam/security-credentials/](http://169.254.169.254/latest/meta-data/iam/security-credentials/)

Karpenter NodePool Manifest (Modern Spot and On-Demand Node Autoscaling)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: dynamic-spot-pool
spec:
  template:
    spec:
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "karpenter.k8s.aws/instance-category"
          operator: In
          values: ["c", "m", "r"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64", "arm64"]
      nodeClassRef:
        name: default-ec2-node-class
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h

5. Gotchas & Common Pitfalls

IMDSv1 Vulnerability & SSRF Exploits: IMDSv1 responds to unauthenticated, simple HTTP GET requests. If an application hosted on the instance contains a Server-Side Request Forgery (SSRF) vulnerability, attackers can fetch http://169.254.169.254 to steal the temporary IAM role credentials assigned to the EC2 instance. Enforce IMDSv2 (HttpTokens=required) across all launch templates.
T-Series CPU Credit Depletion Throttling: When a burstable T-series instance exhausts its CPU credit balance, AWS forcefully throttles CPU execution down to the baseline performance limit (e.g., 10-20% CPU utilization). In production, either migrate to dedicated M or C series instances or enable T Unlimited mode to prevent complete service lockups.
Stop vs. Terminate Cost Dynamics: Stopping an EC2 instance halts vCPU and RAM compute charges immediately; however, all attached Amazon EBS storage volumes, provisioned IOPS, and allocated Elastic IPs continue to generate ongoing billing costs until explicitly released or terminated.

6. Production Best Practices

Eliminate Inbound SSH (AWS Systems Manager Session Manager): Close inbound port 22 in all Security Groups and remove local .pem key pairs. Use AWS SSM Session Manager to connect to instances via encrypted IAM-authenticated sessions, logging all terminal activity to CloudTrail and Amazon S3.
Adopt AWS Graviton (ARM64) Instances: Migrate x86 workloads to ARM64 Graviton instances (c7g, m7g, r7g) to reduce direct instance costs by 20% while gaining up to 40% higher price-performance efficiency.
Deploy Karpenter over Legacy Cluster Autoscaler: Rather than relying on static Auto Scaling Groups that require predefined instance sizes, deploy Karpenter to inspect pending pod specs directly and provision exact-fit Spot/On-Demand instance fleets within seconds.
Configure Automated EC2 Instance Recovery: Set up a CloudWatch Alarm monitoring the StatusCheckFailed_System metric with an attached EC2 Auto-Recovery Action. If the underlying physical hardware hypervisor crashes, AWS automatically reboots and migrates the instance to a healthy physical host while retaining the same instance ID, IP address, and EBS state.

AWS & SRE Field Manual (Part 2): Application (ALB) vs. Network (NLB) Load Balancing Architecture

Enes Guler — Sun, 30 Aug 2026 09:02:10 +0000

1. TL;DR & Problem Statement

Definition: A fully managed, highly available traffic distribution service that automatically routes incoming application and network traffic across multiple targets (Amazon EC2 instances, EKS Pods, AWS Lambda functions, and IP addresses) across multiple Availability Zones.
Problem Solved: Eliminates single-point-of-failure bottlenecks, handles automated health checks, offloads cryptographic TLS termination, and provides scalable ingress routing mechanisms for cloud-native microservices and high-throughput TCP/UDP streams.
Category: Networking & Content Delivery / Ingress Architecture

2. Core Architecture & Key Components

                                Incoming Client Traffic
                                           │
           ┌───────────────────────────────┴───────────────────────────────┐
           ▼                                                               ▼
┌─────────────────────────────────────────┐             ┌─────────────────────────────────────────┐
│ Layer 7: Application Load Balancer (ALB)│             │  Layer 4: Network Load Balancer (NLB)   │
├─────────────────────────────────────────┤             ├─────────────────────────────────────────┤
│ • Inspects HTTP/HTTPS Headers & Paths   │             │ • Operates strictly at TCP/UDP/TLS      │
│ • Path-Based & Host-Based Routing       │             │ • Non-Inspecting Packet Forwarding      │
│ • SSL Termination & Sticky Sessions     │             │ • Ultra-Low Latency (Sub-millisecond)   │
└────────────────────┬────────────────────┘             └────────────────────┬────────────────────┘
                     │                                                       │
        HTTP / HTTPS Ingress                                    Raw TCP/UDP Stream
                     │                                                       │
        ┌────────────┴────────────┐                                          │
        ▼                         ▼                                          ▼
┌──────────────┐          ┌──────────────┐                        ┌────────────────────┐
│ API Pods     │          │ Frontend Pods│                        │ Ingress Controller │
│ (/api/*)     │          │ (/static/*)  │                        │ / Kafka / Games    │
└──────────────┘          └──────────────┘                        └────────────────────┘

2.1. Application Load Balancer (ALB — OSI Layer 7)

Deep Packet Inspection: Evaluates application-layer headers, hostnames, HTTP methods, query parameters, and URL paths.
Content-Based Routing: Routes traffic intelligently across multiple Target Groups (e.g., /api/* to backend microservices, /static/* to object caches or static web pods).
Modern Protocol Support: Natively supports HTTP/2, gRPC, and WebSockets alongside automated HTTP-to-HTTPS redirect rules.
Sticky Sessions (Cookie Affinity): Binds subsequent requests from a specific client to the same backend target instance/pod using encrypted cookies when state is not externalized.

2.2. Network Load Balancer (NLB — OSI Layer 4)

Raw Transport Passthrough: Operates at the transport layer, routing raw TCP, UDP, and TLS connections based exclusively on IP addresses and ports without payload inspection.
Ultra-Low Latency: Delivers sub-millisecond connection handling directly in the data path.
Instantaneous Burst Capacity: Capable of handling millions of requests per second (RPS) and absorbing sudden, massive traffic spikes without requiring manual pre-warming tickets.
Static & Elastic IPs: Provides one static public IP per Availability Zone, making it ideal for client firewalls requiring strict IP whitelisting.

3. Deep Dive Engineering & Architectural Comparison

Feature / Metric	Application Load Balancer (ALB)	Network Load Balancer (NLB)
OSI Layer	Layer 7 (Application)	Layer 4 (Transport)
Protocols	HTTP, HTTPS, HTTP/2, gRPC, WebSockets	TCP, UDP, TLS
Routing Decisions	URL path, Host header, Query params, HTTP method	Source/Destination IP and Port
Latency Profile	Milliseconds (2-10 ms)	Sub-millisecond (< 1 ms)
Traffic Spikes	Scales elastically via DNS over minutes	Instantaneous line-rate scaling
Static IP Support	Dynamic DNS resolution (CNAME required)	Static Elastic IP per Availability Zone
Client IP Identification	Injects `X-Forwarded-For` HTTP header	Client IP Preservation / Proxy Protocol v2
Primary Use Cases	Microservices routing, REST/gRPC APIs, Web apps	High-throughput streaming, Kafka, Game servers, K8s Ingress Entry

4. Advanced Integrations & Ingress Mechanics

AWS Load Balancer Controller (Instance Mode vs. IP Mode)

Instance Mode (target-type: instance): Routes traffic to EC2 host NodePorts. Requires an internal network hop through kube-proxy and iptables/IPVS, adding latency and SNAT overhead.
IP Mode (target-type: ip): Leverages the AWS VPC CNI to route ingress traffic directly from the ALB/NLB to the individual Kubernetes Pod IP. Bypasses kube-proxy entirely, reducing latency and packet manipulation.

Preserving Real Client IP

ALB: Appends the originating client IP to the X-Forwarded-For and X-Forwarded-Proto request headers.
NLB: Because Layer 4 cannot modify HTTP headers, real client IPs are preserved via native Client IP Preservation or by enabling Proxy Protocol v2 (prepends a binary connection header).

Deregistration Delay (Connection Draining)

Prevents inflight HTTP requests from dropping (502 Bad Gateway) when a backend instance or Kubernetes pod is marked for termination.
The load balancer stops sending new connections to the deregistering target and waits for active transactions to complete within deregistration_delay.timeout_seconds (default: 300s, typically tuned to 15–30s in Kubernetes).

Cross-Zone Load Balancing

Enabled by default on ALB; optional on NLB.
Evenly distributes traffic across all registered targets in all enabled Availability Zones regardless of which AZ received the initial network packet, eliminating uneven target group saturation.

5. Practical Notes & Configuration Snippets

Kubernetes Ingress Manifest (ALB in IP Mode with SSL Redirect)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789012:certificate/abc-123
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=20
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /v1
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 8080

Terraform: Network Load Balancer with Cross-Zone Load Balancing

resource "aws_lb" "network_lb" {
  name                             = "prod-streaming-nlb"
  internal                         = false
  load_balancer_type               = "network"
  subnets                          = var.public_subnet_ids
  enable_cross_zone_load_balancing = true
  enable_deletion_protection       = true
}

resource "aws_lb_target_group" "nlb_tg" {
  name        = "tcp-stream-tg"
  port        = 9092
  protocol    = "TCP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  proxy_protocol_v2 = true # Enables Proxy Protocol v2 for client IP pass-through

  health_check {
    protocol            = "TCP"
    port                = "9092"
    interval            = 10
    healthy_threshold   = 3
    unhealthy_threshold = 3
  }
}

6. Gotchas & Common Pitfalls

Subnet Capacity & IP Exhaustion: Each ALB node placed in a public subnet dynamically scales and consumes multiple private IP addresses within that subnet. If your public subnet CIDR is too narrow (e.g., /28), ALB scale-out events will fail under heavy traffic.
Misconfigured Security Group on NLB: By default, NLBs historically did not have associated Security Groups; firewalling occurred strictly at the backend instance level. While security groups on NLBs are now supported, failing to allow client ingress on the backend node security group when using client IP preservation will cause silent packet drops.
Target Group Health Check Latency: If health check intervals and thresholds are set too conservatively (e.g., 30s interval, 5 unhealthy thresholds), dead or hung pods will receive live user traffic for over 2 minutes before the load balancer drops them from the target rotation.

7. Production Best Practices

AWS WAF Integration on ALB: Always attach an AWS WAF (Web Application Firewall) WebACL to public-facing ALBs to block common OWASP Top 10 vulnerabilities, SQL injection, cross-site scripting (XSS), and rate-limit abusive IP addresses at the cloud edge.
Access Logs to Amazon S3: Enable access logging on both ALB and NLB. Load balancers stream structured access logs (client IP, request processing latency, TLS ciphers, backend response codes) directly to an S3 bucket with lifecycle rules for compliance and security forensics.
Match Ingress Controller Pod Disruption Budgets (PDB): Ensure the deregistration_delay on the AWS Load Balancer matches the terminationGracePeriodSeconds and preStop sleep hook of your Kubernetes pods to guarantee zero-downtime rolling deployments.

AWS & SRE Field Manual (Part 1): Zero-Trust Identity & Access Management (IAM)

Enes Guler — Sat, 29 Aug 2026 08:53:29 +0000

📌 TL;DR & Problem Statement

Definition: The centralized security control plane that manages authentication and authorization across all AWS cloud resources. It serves as the native AWS equivalent of Kubernetes RBAC.
Problem Solved: Secures access to infrastructure resources using centralized, auditable, fine-grained, and temporary security credentials instead of distributed, static passwords and access keys.
Category: Security, Identity, & Compliance
Core Rule (Implicit Default Deny): All API requests across AWS are strictly denied by default unless explicitly granted by a matching Allow statement.

🏛️ Core Architecture & Key Components

1. IAM Users (Human Identities & Static Credentials)

Represents discrete individuals interacting with AWS.
Access is provided via console password with mandatory Multi-Factor Authentication (MFA), or programmatic access via static AccessKeyId and SecretAccessKey pairs.
Root Account: The initial, all-powerful identity created upon account registration. It must be locked behind hardware/virtual MFA immediately, stripped of static access keys, and reserved exclusively for emergency break-glass procedures.

2. IAM Groups (Role-Based Membership)

Logical collections of users sharing identical permission requirements (e.g., DevOps-Admins, Data-Engineers).
Attaching permissions directly to individual users is an anti-pattern; policies attach to groups, and users inherit privileges through group membership.

3. IAM Roles & AWS STS (Temporary Credentials & Workload Identity)

An identity abstraction without permanent passwords or static keys, assumed dynamically via the sts:AssumeRole API.
Mirrors the mechanics of Kubernetes ServiceAccounts and EKS IRSA / Pod Identities. Workloads (EC2 instances, Lambda functions, ECS/EKS pods) obtain short-lived, self-expiring tokens generated by the AWS Security Token Service (STS).
Blast Radius Reduction: Even if a compute instance or container is compromised, the leaked STS token expires within minutes to hours, preventing long-term credential exfiltration.

4. IAM Policies (Declarative Permission Documents)

Structured JSON documents defining who can execute specific actions against target resources under designated conditions:
- Effect: Explicit Allow or Deny (an Explicit Deny unconditionally overrides any Allow).
- Action: Targeted API operations (e.g., s3:GetObject, ec2:DescribeInstances).
- Resource: The target Amazon Resource Name (ARN) governed by the statement (e.g., arn:aws:s3:::production-app-bucket/*).
- Condition: Contextual prerequisites required to satisfy the policy (source IP ranges, MFA enforcement, secure transport, resource/principal tags).

⚙️ Deep Dive Engineering & Authorization Mechanisms

Policy Evaluation Logic (Order of Precedence)

Incoming API Request
        │
        ▼
[ Is there an Explicit Deny? ] ─────── YES ────► [ Access Denied ]
        │
        NO
        ▼
[ Does SCP allow it? ] ─────────────── NO  ────► [ Access Denied ]
        │
       YES
        ▼
[ Does Resource Policy allow it? ] ─── YES ──┐
        │                                    │
        NO                                   │
        ▼                                    │
[ Does Identity Policy allow it? ] ─── YES ──┼──► [ Check Boundaries/Session ] ──► [ Access Granted ]
        │                                    │
        NO                                   │
        ▼                                    │
 [ Access Denied (Implicit Deny) ] ◄─────────┘

Is there an Explicit Deny anywhere in the evaluation chain? (If yes, immediate termination and rejection).
Does an AWS Organizations Service Control Policy (SCP) permit the request?
Does a Resource-based Policy (e.g., S3 Bucket Policy, KMS Key Policy) allow the action?
Does an Identity-based Policy (IAM Policy attached to the principal) allow the action?
Does the request fall within the ceiling defined by Permissions Boundaries and Session Policies?
Outcome: If any evaluation point yields an Explicit Deny, or if no policy grants an explicit Allow, the request is dropped (Implicit Deny).

IAM Permissions Boundaries

Defines the maximum permission ceiling for an IAM entity.
Enables delegating policy creation to development teams without risking privilege escalation. Even if a developer creates a new role with AdministratorAccess, the effective permissions will never exceed the attached Permissions Boundary (e.g., S3FullAccessBoundary).

Attribute-Based Access Control (ABAC) & Session Tags

Grants dynamic access based on matching metadata tags rather than maintaining complex, static JSON policies per user.
A single generic policy allows access if the principal's department or project tag matches the target resource tag (aws:ResourceTag/Project == aws:PrincipalTag/Project).

Cross-Account Access & ExternalID (Confused Deputy Protection)

When delegating role access to third-party SaaS vendors (e.g., Datadog, Databricks), requiring an sts:ExternalId condition inside the Role Trust Policy is mandatory.
Prevents the Confused Deputy attack vector, where an adversary manipulates the third-party platform into assuming another customer's role inside your AWS account.

💻 Practical Notes & Configuration Snippets

Attribute-Based Access Control (ABAC) Policy (Tag-Match Enforcement)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Project": "${aws:PrincipalTag/Project}"
        }
      }
    }
  ]
}

Essential AWS CLI Commands for IAM & STS

# Check current caller identity and active ARN
aws sts get-caller-identity

# Assume an IAM Role manually (returns temporary credentials)
aws sts assume-role \
  --role-arn "arn:aws:iam::123456789012:role/DevOpsAdminRole" \
  --role-session-name "CLI-Session" \
  --external-id "UniqueSecretId123"

# List IAM policies attached to a specific group
aws iam list-attached-group-policies --group-name "Engineering-Team"

⚠️ Gotchas & Common Pitfalls

Eventual Consistency in IAM APIs: IAM is a globally distributed control plane. Changes to policies, roles, or group memberships take several seconds to propagate across all AWS edge endpoints worldwide. Automation scripts and CI/CD pipelines must incorporate exponential backoff or brief delay retries immediately following IAM resource provisioning.
Identity-Based vs. Resource-Based Policy Overlaps: An Identity-based policy is a credential held by the caller; a Resource-based policy (e.g., S3 Bucket Policy, SQS Policy) is an access control list on the target resource. Within the same AWS account, if a resource-based policy grants explicit access, the caller can execute the action even without an attached identity-based IAM policy.
Wildcard (*) Operator Abuse: Using Action: "" and Resource: "" violates the Principle of Least Privilege and represents the primary security vulnerability identified during compliance audits.

💡 Production Best Practices

Enforce IMDSv2 Globally: Prevent SSRF-driven IAM token leakage from compute nodes by enforcing IMDSv2 (HttpTokens=required, HttpPutResponseHopLimit=1) across all EC2 launch templates and running instances.
Automate IAM Access Analyzer: Enable IAM Access Analyzer across the organization to automatically flag external, cross-account, and public access to S3 buckets, KMS keys, and IAM roles.
Root Account Monitoring & Alarms: Deploy an Amazon EventBridge rule tied to SNS alerts to notify security teams immediately via Slack or PagerDuty whenever the Root user logs into the AWS Management Console or triggers an API call.
Service Control Policies (SCPs) as Hard Guardrails: Implement SCPs at the AWS Organizations root to enforce organizational compliance (e.g., restricting available AWS Regions or denying modification to security audit log buckets). SCP Deny statements cannot be bypassed by any user within a member account, including IAM identities with full AdministratorAccess.

Kubernetes Architecture Deep Dive: From Resource Limits to Custom Operators

Enes Guler — Wed, 12 Aug 2026 12:33:54 +0000

Resource Management (Requests & Limits)

Resource management prevents Kubernetes workloads from depleting node hardware or causing resource contention among containers.

Resource Requests

The absolute minimum CPU and Memory guaranteed for a Pod to start. The Kubernetes Scheduler uses requests to determine node placement. If a node cannot fulfill the requested resources, the Pod will not be scheduled on that node.

Architectural Takeaway: Omitting requests leads to poor scheduling decisions, resulting in unbalanced cluster distribution and potential node starvation.

Resource Limits

The maximum ceiling of CPU and Memory a Pod is allowed to consume.

Memory (Non-Compressible Resource): If a process (e.g., a memory-heavy Pandas pipeline) exceeds its memory limit by even 1 MB, the Linux Kernel terminates the container with an OOMKilled (Out Of Memory Killed) exit code. This acts as a critical safety circuit breaker to constrain the blast radius and protect co-located services.
CPU (Compressible Resource): Unlike memory, exceeding CPU limits does not terminate the pod. Instead, the Linux Completely Fair Scheduler (CFS) enforces CPU Throttling. This constrains CPU usage, keeping the application alive but causing severe latency spikes during heavy traffic.

Architectural Takeaway: Memory limits protect nodes from crashing due to leaks, while improperly tuned CPU limits risk performance degradation via throttling even when the host node has idle CPU capacity.

Advanced Architectural Concepts

Beyond basic resource allocation, managing production Kubernetes clusters requires an understanding of hardware overcommitment, Linux Kernel throttling mechanics, and implicit Quality of Service (QoS) eviction hierarchies.

+-----------------------------------------------------------------------------------+
|                            Node Hardware Capacity                                 |
|                                                                                   |
|  +---------------------------+  +---------------------------+  +---------------+  |
|  | Guaranteed Pod            |  | Burstable Pod             |  | BestEffort    |  |
|  | Requests == Limits        |  | Requests < Limits         |  | No Req/Limits |  |
|  | (Lowest OOM Kill Priority)|  | (Medium Eviction Risk)    |  | (First Killed)|  |
|  +---------------------------+  +---------------------------+  +---------------+  |
+-----------------------------------------------------------------------------------+

Resource Overcommit

Overcommit occurs when the sum of all container resource limits on a node exceeds the node's actual physical hardware capacity, while the sum of resource requests remains within capacity bounds.

The Economics: In cloud infrastructure, running compute nodes at 20% average utilization is an expensive waste. Overcommit allows engineering teams to pack more workloads onto fewer nodes by betting that not all pods will hit their maximum resource limits simultaneously.
The Overcommit Trade-off:
- CPU Overcommit: Safe and manageable. CPU is a compressible resource; if demand exceeds capacity, execution speed slows down across pods.
- Memory Overcommit: High risk. Memory is non-compressible. If multiple pods suddenly spike toward their memory limits concurrently, the node runs out of physical RAM and swap. The Linux Kernel triggers OOMKilled events to forcibly terminate containers and reclaim memory.
Architectural Takeaway: Calculate the Overcommit Ratio (Sum of Limits) / (Physical Capacity) carefully. Overcommit CPU aggressively to save money, but keep memory overcommit conservative to avoid cascading application crashes.

The Linux CFS Quota & CPU Throttling Trap

Setting CPU limits relies on the Linux Kernel Completely Fair Scheduler (CFS) using cgroup enforcement. The Kernel evaluates CPU usage in enforced time windows, typically every 100ms (the CFS Period).

How the Trap Works: If a pod with a CPU limit of 1 vCPU (1000m) executes a multi-threaded operation that consumes 100ms worth of CPU processing time within the first 20ms of a period, the Kernel locks out the pod's CPU access for the remaining 80ms of that window.

100ms CFS Period Window
┌────────────────────────┬─────────────────────────────────────────┐
│  Multi-Threaded Burst  │          CPU Throttled / Locked         │
│   (Consumes 100ms CPU) │           (Latency Spikes Hit)          │
└────────────────────────┴─────────────────────────────────────────┘
 0ms                    20ms                                     100ms

The Idle Node Paradox: A pod can experience severe CPU Throttling (causing 500ms+ latency spikes in HTTP services) even when the underlying host worker node shows 80% idle CPU capacity.
Architectural Takeaway: Many enterprise SRE teams disable CPU limits entirely (Limits: Unset) for latency-sensitive microservices, relying strictly on well-tuned CPU Requests paired with Horizontal Pod Autoscaler (HPA) to handle traffic spikes safely.

Quality of Service (QoS) Classes & Eviction Order

Kubernetes automatically assigns every pod a QoS Class based on how its container requests and limits are configured. When a worker node experiences memory pressure, the Linux Kernel and Kubernetes Kubelet use the QoS class to determine eviction priority via the oom_score_adj metric.

Node Resource Exhaustion (Eviction Sequence)
────────────────────────────────────────────────────────────────────────►
[BestEffort Pods]   ───►   [Burstable Pods]   ───►   [Guaranteed Pods]
 (Terminated First)         (Terminated Second)       (Protected / Last)

Guaranteed (Highest Priority / Protected)
- Condition: Every container in the pod must explicitly specify both CPU and Memory, and Requests == Limits for all resources.
- Behavior: Highly stable. Granted an oom_score_adj of -997. These pods are the absolute last to be evicted or terminated during node memory starvation. Ideal for databases, core stateful sets, and critical payment services.
Burstable (Medium Priority / Standard)
- Condition: At least one container specifies a request or limit, but Requests != Limits (or CPU has a limit while Memory does not).
- Behavior: Allowed to burst beyond baseline when capacity allows. Evicted after all BestEffort pods are killed if memory pressure persists. Ideal for web APIs, background workers, and standard web applications.
BestEffort (Lowest Priority / Disposable)
- Condition: No requests or limits are defined for any container in the pod.
- Behavior: Assigned an oom_score_adj of 1000. Gets access to unallocated node resources, but is the first target for termination during node memory pressure. Ideal for non-critical batch processing, dev/test pods, or temporary log collectors.

Security and Observability

The control mechanisms that keep the system resilient before everything crashes or when a breach occurs.

RBAC (Role-Based Access Control)

The identity card and permission firewall for your code. It strictly limits what a Pod (or a user) can execute within the system.

ServiceAccount: The identity assigned to a Pod.
Role/ClusterRole: The explicit list of allowed permissions.
RoleBinding/ClusterRoleBinding: The bridge that staples an identity to a set of permissions. Even if a vulnerability leaks into the application code, RBAC prevents lateral movement and protects the underlying infrastructure.

Monitoring & Alerting

Kubernetes knows whether an application is running, but it is blind to the question: “Is the business logic actually behaving correctly?” To solve this:

Prometheus: Continuously scrapes and collects every metric (CPU, API response times, queue lengths, etc.) from the system.
Grafana: Transforms raw time-series metrics into visual, customizable dashboards.
Alertmanager: Triggers notifications (Slack, PagerDuty, Email) when response latency spikes from 200ms to 4 seconds, or when OOMKilled errors start popping up.

Service Mesh (Istio/Linkerd)

An infrastructure highway that manages, encrypts, and observes inter-service communication (East-West Traffic).

Sidecar Proxy: A lightweight proxy (like Envoy or NGINX) injected alongside your main application container inside the same Pod. It intercepts all inbound and outbound traffic completely transparently to your application code.
mTLS (Mutual TLS): Automatically encrypts network traffic between microservices in transit. Even if an attacker intercepts the network traffic, the payload remains unreadable.
Traffic Splitting: Allows you to route a portion of live traffic (e.g., 10%) to a new application version (Canary deployment) without altering a single line of application code.

Pod Lifecycle and Probes

Kubernetes evaluates the true health and readiness of an application inside a Pod using three distinct types of Probes.

Startup Probe

Ensures Kubernetes remains patient while an application boots up (for instance, legacy Java applications that might take over a minute to initialize). No other probes run until the Startup Probe succeeds. If this probe fails, Kubernetes assumes the application is stuck during startup and immediately restarts the Pod.

Liveness Probe

Answers the question: “Is this Pod alive?” If the application enters a deadlock or infinite loop, Kubernetes detects the stall via the probe, terminates the unresponsive container, and restarts it according to the restart policy.

Readiness Probe

Answers the question: “This Pod is alive, but is it ready to accept incoming user requests?” For example, the application process might be running, but it’s still establishing connection to a database. If this probe fails, Kubernetes does NOT kill or restart the Pod; it simply removes the Pod’s IP from the Service endpoints to stop routing traffic to it. Once the connection is established and the probe passes, traffic automatically resumes.

Networking Architecture: Ingress vs. Service

The networking layer that defines how traffic flows within the cluster and how external users access internal workloads.

ClusterIP (Service)

An internal extension line that allows Pods to communicate with one another within the cluster. It is completely isolated from the outside world.

NodePort (Service)

Exposes a specific port on every Node directly to the public internet. It is insecure and inefficient, making it rarely suitable for production environments.

LoadBalancer (Service)

Provisions a dedicated, paid external Cloud Load Balancer from your cloud provider (e.g., AWS ALB/NLB). Provisioning a separate cloud load balancer for every single microservice drastically inflates your cloud bill.

Ingress

A single Smart Traffic Router (Reverse Proxy) positioned at the cluster gateway. It terminates traffic from a single external Load Balancer and intelligently routes incoming requests to internal services based on domain names or URL path rules (/api, /auth). It minimizes infrastructure costs while providing a centralized point for TLS termination and traffic management.

Smart Placement

It is the set of rules that decides which applications run on which physical/virtual servers.

Taints

A security barrier (or label) applied to a Node. It is a node’s way of saying: “I am an expensive, GPU-heavy server. Standard web APIs without the right **tolerance* should stay away from me.”*

Tolerations

A specification written inside a Pod’s configuration. It is a Pod’s way of saying: “Yes, that server has a GPU taint, but I have the tolerance for it. You can place me there.”

Node Affinity

A specific requirement whispered by a Pod to the Kubernetes Control Plane. It is a Pod’s way of saying: “Do not place me randomly; run me strictly (or preferably) on Memory-Optimized nodes.”

Hard Constraint (required...): “I MUST have this node type, or do NOT deploy me at all.” (Strict rule)
Soft Constraint (preferred...): “I WOULD LIKE this node type, but if it’s not available, just put me anywhere.” (Preference)

Scaling and Autonomy

A set of mechanisms used to balance performance and infrastructure costs as workload demand increases or decreases.

HPA (Horizontal Pod Autoscaler)

Monitors CPU and RAM usage of Pods. If a Pod is under heavy load (heats up), HPA increases the replica count; as load drops (cools down), it scales back down. It is inherently sluggish because waiting for CPU metrics to spike takes time.

KEDA (Kubernetes Event-Driven Autoscaling)

Grants Kubernetes the ability to listen to external event sources (such as Redpanda, Kafka, S3, RabbitMQ, etc.). If a queue in Redpanda is empty, KEDA can scale the Pod replicas down to zero (Scale-to-Zero) to completely eliminate compute costs. If 10,000 messages suddenly hit the queue, it immediately spins up 50 Pods within seconds without waiting for CPU usage to rise.

Data and Storage

The layer that overcomes the ephemeral (temporary) nature of containers to manage persistent data.

CSI (Container Storage Interface)

The standard interface Kubernetes uses to communicate with cloud providers regarding storage. When a Pod needs a 50 GB disk for a database or AI model weights, the CSI driver dynamically provisions that disk from the cloud and physically attaches it to the node hosting the Pod. If the Pod dies and reschedules onto another node, CSI detaches the disk and reattaches it to the new node.

PV (Persistent Volume)

The physical/actual disk allocated to the cluster by the infrastructure administrator. It is a cluster-level resource independent of Pods, representing a declaration like: “I have a 100 GB SSD-backed disk available in AWS.”

PVC (Persistent Volume Claim)

A request for storage made by a developer (or application) to Kubernetes. It is a Pod’s way of saying: “I urgently need a 20 GB high-speed read/write disk.”

How PV and PVC Interact

When a developer creates a PVC, Kubernetes acts like a matchmaker. It scans the pool of available PVs. If it finds a PV that meets the requested capacity and access modes, it binds that PVC to the PV. The Pod only knows the name of the PVC and doesn’t care about the underlying storage implementation.

SC (StorageClass)

The set of instructions given to the CSI driver. When a developer specifies storageClassName: gp3 inside a PVC, Kubernetes checks if a matching PV exists. If no pre-provisioned PV is available, it immediately triggers the CSI driver. The CSI driver then provisions a 20 GB volume on AWS, registers it in Kubernetes as a new PV, and binds it to the PVC. This entire automated workflow is called Dynamic Provisioning.

Teaching K8s a New Language (Extending Kubernetes)

The capability that transforms Kubernetes from a simple application runner into a customizable platform builder.

CRD (Custom Resource Definition)

The process of extending Kubernetes’ built-in dictionary (which natively includes objects like Pods, Services, and Deployments) with your own custom resources. You define and register brand-new objects—such as ModelDeployment or PostgresDatabase—directly into the Kubernetes API database (etcd).

Operator Pattern

Kubernetes natively recognizes custom resources like ModelDeployment, but it doesn’t know what actions to take when one is created. An Operator is a custom controller written in languages like Go or Python that hooks into the Kubernetes control loop. When someone submits a ModelDeployment YAML manifest, the Operator wakes up and says: “Got it! I need to pull the AI model weights from S3, schedule a GPU-enabled node, and expose a REST API.” It then automatically orchestrates all the underlying standard Kubernetes objects on your behalf.

DEV Community: Enes Guler

Terraform & IaC Field Manual (Part 1): Core Architecture, State Locking & Lifecycle Engineering

1. Core Architectural Paradigms

1.1. Declarative vs. Imperative Paradigms

1.2. Mutable vs. Immutable Infrastructure Strategy

1.3. Provisioning vs. Configuration Management Layering

2. Core Architecture & Provider Decoupling

2.1. Terraform Core

2.2. Providers (Plugin Ecosystem)

2.3. Provider Resolution, Locking, and Optimization

1. Provider Source & Version Constraints

2. Dependency Lock File (.terraform.lock.hcl)

3. Plugin Caching for CI/CD Pipelines

4. Multiple Provider Configurations (Provider Aliases)

3. State Management: The Source of Truth

3.1. Critical Risks Associated with State Files

3.2. Remote Backends & State Locking

3.3. Production S3 Backend Hardening Checklist

3.4. State CLI Operations & Emergency Management

3.5. State Isolation: Directory-Based vs. Workspace Isolation

4. Terraform Execution Lifecycle Commands

4.1. Core Command Breakdown

4.2. Production CI/CD Pattern: Deterministic Plans

4.3. Targeted & Emergency Execution Flags

5. Dependency Resolution: Implicit vs. Explicit Dependencies

5.1. Implicit Dependencies (Attribute References)

5.2. Explicit Dependencies (depends_on Meta-Argument)

5.3. Module-Level Dependencies

5.4. Dependency Management Best Practices

6. Resource Lifecycle Rules: Production Safety Mechanisms

6.1. Zero-Downtime Replacement (create_before_destroy)

6.2. Accidental Teardown Protection (prevent_destroy)

6.3. Dynamic Drift Suppression (ignore_changes)

6.4. Conditional Triggers & Custom Assertions

6.5. Comprehensive Production Resource Template

AWS & SRE Field Manual (Part 9): SRE Framework — SLI, SLO, SLA & Error Budget Engineering

1. TL;DR & Problem Statement

2. Core Architecture & Key Components

2.1. SLI (Service Level Indicator) - Empirical & Raw Telemetry

2.2. SLO (Service Level Objective) - Internal Engineering Target

2.3. SLA (Service Level Agreement) - External Customer Contract

3. Deep Dive Engineering & Operational Matrix

4. Advanced SRE Mechanisms & Reliability Patterns

Multi-Window Multi-Burn-Rate Alerting (Google SRE Standard)

Composite SLIs & User Journey Mapping

The Over-Reliability Paradox & Chaos Injection

5. Practical Notes & Configuration Snippets

PromQL: Calculating 5-Minute Rolling HTTP Availability SLI

OpenSLO Specification Manifest (SLO as Code)

6. Gotchas & Common Pitfalls

7. Production Best Practices

AWS & SRE Field Manual (Part 8): Amazon CloudWatch Architecture, Telemetry Pipelines & Production Observability

1. TL;DR & Problem Statement

2. Core Architecture & Key Components

2.1. CloudWatch Logs (Log Groups & Streams)

2.2. CloudWatch Logs Insights (Interactive Log Analytics)

2.3. CloudWatch Metrics (System vs. Custom Telemetry)

2.4. CloudWatch Alarms & Composite Alarms

3. Deep Dive Engineering & Advanced Patterns

Embedded Metric Format (EMF) & High Cardinality

Container Insights with AWS Distro for OpenTelemetry (ADOT)

Metric Math (Derived Expressions)

4. Practical Notes & Configuration Snippets

CloudWatch Logs Insights: Aggregating API 5xx Errors by Endpoint

Embedded Metric Format (EMF) Structured Log Output

Terraform: Composite Alarm Definition

5. Gotchas & Common Pitfalls

6. Production Best Practices

AWS & SRE Field Manual (Part 7): Amazon RDS Deep Dive: Architecture and High Availability

1. TL;DR & Problem Statement

2. Core Architecture & Key Components

2.1. Multi-AZ High Availability (Synchronous DR)

2.2. Horizontal Read Scaling (Read Replicas)

2.3. Amazon Aurora Architecture (Decoupled Compute & Storage)

2.4. Amazon RDS Proxy (Managed Connection Pooler)

3. Deep Dive Engineering & Architecture Patterns

Replication Lag & Read-After-Write Consistency

Automated Backups & Point-in-Time Recovery (PITR)

4. Practical Notes & Configuration Snippets

Terraform: PostgreSQL RDS Instance with Multi-AZ & Storage Autoscaling

2. Dependency Lock File (`.terraform.lock.hcl`)

5.2. Explicit Dependencies (`depends_on` Meta-Argument)

6.1. Zero-Downtime Replacement (`create_before_destroy`)

6.2. Accidental Teardown Protection (`prevent_destroy`)

6.3. Dynamic Drift Suppression (`ignore_changes`)

Kubernetes `volumeBindingMode: WaitForFirstConsumer`

Production Kubernetes StorageClass (`gp3` with Delayed Binding)

Terraform: Provisioning an EBS `gp3` Volume with Custom IOPS & Throughput