DEV Community: AlpeshKumbhare

Migrating to AWS: The 7 Rs, Migration Tools, and How to Actually Move an Enterprise Portfolio

AlpeshKumbhare — Mon, 07 Sep 2026 06:47:42 +0000

Migrating a handful of servers to AWS is easy. Migrating an enterprise portfolio of hundreds of applications — with dependencies, compliance constraints, and zero downtime tolerance — is a program, not a project. The difference between success and a stalled migration is a clear strategy per application and the right tools for each phase.

This guide covers the 7 Rs migration framework, the migration lifecycle, and the AWS tools that turn "we should move to cloud" into a tracked, executable plan.

The Migration Lifecycle

┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
│  ASSESS   │──→│ MOBILIZE │──→│ MIGRATE  │──→│MODERNIZE │
│          │   │          │   │          │   │          │
│ Discover  │   │ Landing  │   │ Move     │   │ Optimize │
│ portfolio │   │ zone,    │   │ workloads│   │ refactor │
│ + business│   │ pilot,   │   │ in waves │   │ to cloud │
│ case      │   │ skills   │   │          │   │ native   │
└──────────┘   └──────────┘   └──────────┘   └──────────┘

Most failed migrations skip Assess (no dependency mapping) or Mobilize (no landing zone) and jump straight to moving servers. Don't.

The 7 Rs: Strategy Per Application

Every application gets one of seven dispositions. This is the core decision.

┌────────────────────────────────────────────────────────────────┐
│  7 Rs MIGRATION STRATEGIES                                       │
├────────────┬─────────────────────────────────────────────────────┤
│ Retire      │ Decommission — no longer needed                     │
│ Retain      │ Keep on-premises (for now) — not ready/not worth it │
│ Relocate    │ Move VMware as-is (VMware Cloud on AWS / EVS)        │
│ Rehost      │ Lift-and-shift (MGN) — move, then optimize          │
│ Replatform  │ Lift-tinker-shift (e.g., DB → RDS, self-mgd → managed)│
│ Repurchase  │ Drop and shop (move to SaaS)                        │
│ Refactor    │ Re-architect for cloud-native (microservices, serverless)│
└────────────┴─────────────────────────────────────────────────────┘

Choosing the Right R

Strategy	Effort	When to Use	Example
Retire	None	App is unused or redundant	Legacy reporting tool nobody opens
Retain	None	Compliance/latency keeps it on-prem, or mainframe not ready	Regulated system pending modernization
Relocate	Low	VMware estate, want fast exit from data center	vSphere workloads → EVS / VMware Cloud
Rehost	Low	Speed matters, optimize later	App server → EC2 via MGN
Replatform	Medium	Small changes yield big managed-service wins	Self-managed MySQL → RDS/Aurora
Repurchase	Medium	Commercial SaaS exists	On-prem CRM → Salesforce
Refactor	High	Strategic app needing scale/agility	Monolith → microservices on ECS/Lambda

The Pragmatic Reality

Most large migrations are 70% Rehost/Relocate, 20% Replatform, 10% Refactor. You cannot refactor everything — it's too slow and expensive. Rehost first to exit the data center (hit the deadline), then modernize the applications that justify the investment. This is the "move then improve" pattern.

Phase 1: Assess

Discovery — You Can't Migrate What You Can't See

AWS Application Discovery Service collects server configs, performance metrics, and — critically — network dependencies from your on-premises environment.

Two collection methods:

Method	How	Best For
Agentless (OVA appliance)	VM in vCenter, collects VM inventory	VMware environments
Agent-based	Install agent on each server	Detailed process + dependency data

Discovery answers: What do we have? How is it used? What talks to what? (Dependency mapping prevents the classic "we migrated the app but forgot it calls a database still on-prem" disaster.)

Migration Evaluator / Business Case

Before moving anything, build the business case:

Current TCO (hardware, licensing, data center, staff)
Projected AWS cost (right-sized, with Savings Plans)
Migration cost + timeline
Risk assessment per application

Phase 2: Mobilize

Build the Foundation

Before migrating workloads, establish:

Landing Zone — multi-account structure, guardrails, networking (Control Tower / Landing Zone Accelerator)
Network connectivity — Direct Connect or Site-to-Site VPN to on-premises
Security baseline — GuardDuty, Security Hub, IAM Identity Center
Migration pilot — migrate 1-3 low-risk applications to validate the process and build team skills

Migration Waves

Group applications into waves based on:

Dependencies — migrate tightly-coupled apps together
Complexity — start simple, build confidence
Business criticality — non-critical first, crown jewels later
Team capacity — realistic parallel workload

Phase 3: Migrate — The Tools

AWS Migration Hub: Single Pane of Glass

Migration Hub is the command center — it tracks migration status across all tools and applications in one dashboard.

                    ┌─────────────────────┐
                    │   Migration Hub      │
                    │  (tracks everything) │
                    └──────────┬──────────┘
              ┌────────────────┼────────────────┐
              │                │                │
              ▼                ▼                ▼
      ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
      │ App Discovery│ │     MGN      │ │     DMS      │
      │ (assess)     │ │  (rehost)    │ │ (databases)  │
      └──────────────┘ └──────────────┘ └──────────────┘

AWS Application Migration Service (MGN) — for Rehost

The primary lift-and-shift tool. Continuous block-level replication from source servers to AWS:

Install MGN agent on source servers (or agentless for vCenter)
Continuous replication to a staging area in AWS (low-cost)
Launch test instances — validate without affecting source
Cutover — final sync, launch production instances, redirect traffic

Key benefit: Minimal downtime (minutes at cutover), continuous sync means you can test repeatedly before committing.

AWS Database Migration Service (DMS) — for Databases

Migrates databases with minimal downtime, including heterogeneous migrations (Oracle → PostgreSQL, SQL Server → Aurora):

Homogeneous (same engine): direct migration
Heterogeneous (different engine): use Schema Conversion Tool (SCT) first to convert schema/code, then DMS for data
CDC (Change Data Capture): keeps source and target in sync during migration for near-zero downtime

Tool Selection by Strategy

Strategy	Primary Tool
Rehost	Application Migration Service (MGN)
Relocate (VMware)	VMware Cloud on AWS / Amazon EVS
Replatform (database)	DMS + Schema Conversion Tool
Refactor	Manual re-architecture + CI/CD
Mainframe	AWS Mainframe Modernization service

Phase 4: Modernize

Once workloads are on AWS, modernize the ones that justify it:

From	To	Benefit
EC2 app servers	Containers (ECS/EKS) or Lambda	Scaling, cost, agility
Self-managed databases	RDS/Aurora/DynamoDB	Managed ops, auto-scaling
Monolith	Microservices	Independent deployment, scaling
Batch on EC2	Step Functions + Lambda	Serverless, pay-per-use
File shares	S3 + EFS/FSx	Durability, cost tiers
Data warehouse	Redshift / Lakehouse (Iceberg)	Modern analytics

AI-Assisted Modernization (2026)

AWS now offers AI-agent-driven modernization — particularly for mainframes. Specification-driven approaches use AI agents to analyze legacy code (COBOL), extract business logic, and generate modern microservices. This dramatically reduces the cost of the hardest migrations.

Migration Patterns

Pattern 1: The Strangler Fig (Gradual Refactor)

Instead of a big-bang rewrite, incrementally replace pieces of a monolith:

Legacy Monolith
    │
    ├── Route new feature → New microservice (AWS)
    ├── Route feature B → New microservice (AWS)
    └── (monolith shrinks over time until retired)

New functionality goes to cloud-native services; the monolith gradually "strangled" until it's gone. Lower risk than big-bang.

Pattern 2: Rehost Then Replatform

On-prem → MGN rehost to EC2 (hit the data center exit deadline)
       → then: migrate DB to RDS (replatform)
       → then: containerize app (modernize)

Get out of the data center fast, improve incrementally. Avoids the trap of trying to perfect everything before moving.

Pattern 3: Database-First (for data-gravity apps)

1. DMS replicate database to AWS (CDC keeps in sync)
2. Migrate application servers (MGN)
3. Cutover: point apps to AWS database, stop CDC

Common Migration Mistakes

Mistake	Impact	Fix
No dependency mapping	Break apps that call un-migrated services	Application Discovery Service first
Refactor everything	Migration never finishes, budget blown	Rehost 70%, refactor selectively
No landing zone first	Ungoverned sprawl, security gaps	Build landing zone in Mobilize phase
Lift-and-shift then stop	Pay for un-optimized EC2 forever	Plan modernization backlog post-migration
Big-bang cutover	High-risk, hard to roll back	Wave-based migration with pilots
Ignoring licensing	Bring problematic licenses to cloud	Assess BYOL vs license-included
No rollback plan	Stuck if cutover fails	MGN test instances + documented rollback

The Migration Readiness Checklist

ASSESS
☐ Application Discovery Service deployed
☐ Dependency mapping complete
☐ 7 Rs disposition assigned per app
☐ Business case / TCO analysis done

MOBILIZE
☐ Landing zone deployed (Control Tower / LZA)
☐ Direct Connect / VPN to on-premises
☐ Security baseline (GuardDuty, Security Hub)
☐ Pilot migration completed + lessons captured
☐ Migration waves planned

MIGRATE
☐ Migration Hub tracking configured
☐ MGN for rehost workloads
☐ DMS + SCT for database migrations
☐ Test instances validated before cutover
☐ Rollback procedure documented per wave

MODERNIZE
☐ Post-migration optimization backlog
☐ Right-sizing (Compute Optimizer)
☐ Savings Plans purchased
☐ Modernization candidates prioritized

Summary

Migrating an enterprise to AWS is a disciplined program:

Assess — Discover the portfolio, map dependencies, assign a strategy (one of the 7 Rs) per app, build the business case.
Mobilize — Landing zone, connectivity, security baseline, pilot migration, wave planning.
Migrate — Migration Hub as command center; MGN for rehost, DMS+SCT for databases, EVS for VMware.
Modernize — Optimize and refactor the applications that justify the investment.

The strategic truth: You can't refactor everything, and you shouldn't try. Rehost the bulk to exit the data center on time, then modernize selectively where it delivers business value. "Move then improve" beats "perfect then move" every time.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in cloud migration and AWS infrastructure automation. Connect on LinkedIn.

AWS Secrets Management: Secrets Manager vs Parameter Store vs KMS — The Complete Decision Guide

AlpeshKumbhare — Thu, 03 Sep 2026 08:36:16 +0000

Secrets management is one of those decisions you make on day zero and regret in month six if you get it wrong. Pick the wrong store and you'll either rewrite your bootstrap code later or pay thousands a year for configuration that could have been stored for free.

AWS gives you two primary stores — Secrets Manager and Systems Manager Parameter Store — plus KMS underpinning both. They overlap enough to be confusing but differ enough that the choice matters. This guide breaks down when to use each, with cost, rotation, and access patterns.

The Core Distinction

┌──────────────────────────────────────────────────────────────────┐
│  SECRETS MANAGER          │  PARAMETER STORE (SSM)                 │
│                           │                                        │
│  Purpose-built for        │  General-purpose config store          │
│  SECRETS that rotate      │  that ALSO holds secrets               │
│                           │                                        │
│  • Automatic rotation     │  • Free standard tier                  │
│  • Cross-region replica   │  • Config + secrets together           │
│  • Cross-account sharing  │  • Hierarchical paths                  │
│  • ~$0.40/secret/month    │  • SecureString via KMS                │
│  • Built-in DB integration│  • No native rotation                  │
└──────────────────────────────────────────────────────────────────┘

One-line rule: If it needs to rotate, use Secrets Manager. If it's configuration (even encrypted config), use Parameter Store.

Quick Decision Matrix

Requirement	Use
Database credentials with auto-rotation	Secrets Manager
API keys that must rotate on schedule	Secrets Manager
Cross-region secret replication	Secrets Manager
Cross-account secret sharing	Secrets Manager
Application config (URLs, feature flags)	Parameter Store (Standard)
Static secrets that rarely change	Parameter Store (SecureString)
Free tier / cost-sensitive	Parameter Store (Standard)
Large parameters (up to 8KB)	Parameter Store (Advanced)
Encryption keys themselves	KMS

AWS Secrets Manager

Purpose-built for secrets that need lifecycle management — rotation, replication, and controlled sharing.

Key Capabilities

Feature	Detail
Automatic rotation	Lambda-driven rotation on a schedule (built-in support for RDS, Aurora, Redshift, DocumentDB)
Cross-region replication	Replicate secrets to other regions for DR and low-latency access
Cross-account access	Resource policies allow sharing secrets across accounts
Version staging	AWSCURRENT / AWSPENDING / AWSPREVIOUS labels for safe rotation
KMS encryption	All secrets encrypted with KMS (customer-managed or AWS-managed key)
Fine-grained IAM	Control access per secret via IAM + resource policies

Automatic Rotation

This is the killer feature. For supported databases, Secrets Manager handles the entire rotation dance:

Rotation schedule triggers (e.g., every 30 days)
    │
    ▼
Lambda rotation function:
  1. Create new credential (AWSPENDING)
  2. Update credential in the database
  3. Test new credential works
  4. Mark new credential AWSCURRENT
  5. Old credential → AWSPREVIOUS (grace period)

Applications always request AWSCURRENT — they get the latest valid credential without knowing rotation happened.

Retrieving a Secret

import boto3, json

client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId='prod/db/credentials')
secret = json.loads(response['SecretString'])
# Use secret['username'], secret['password']

Best practice: Cache the secret (SDK caching library or Lambda extension) — don't call get_secret_value on every request. It's an API call with cost and latency.

Cost

$0.40 per secret per month
$0.05 per 10,000 API calls

At 100 secrets, that's ~$40/month before API calls. This adds up — which is why you shouldn't put non-rotating config here.

Systems Manager Parameter Store

A general-purpose hierarchical store for configuration and secrets. The free standard tier makes it the default home for anything that doesn't need rotation.

Parameter Types

Type	Encrypted	Use For
String	No	Plain config (URLs, region names, feature flags)
StringList	No	Comma-separated lists
SecureString	Yes (KMS)	Secrets that don't need auto-rotation

Standard vs Advanced Tier

Feature	Standard	Advanced
Cost	Free	$0.05/parameter/month
Max parameters	10,000	100,000
Max value size	4 KB	8 KB
Parameter policies (expiration, notification)	❌	✅
Higher throughput	❌	✅

Hierarchical Organization

Parameter Store shines with path-based hierarchies:

/myapp/prod/db/host
/myapp/prod/db/port
/myapp/prod/api/endpoint
/myapp/staging/db/host
/myapp/dev/db/host

Fetch an entire branch with one call:

import boto3

ssm = boto3.client('ssm')
response = ssm.get_parameters_by_path(
    Path='/myapp/prod/',
    Recursive=True,
    WithDecryption=True
)
# Returns all prod parameters in one call

Retrieving a Parameter

import boto3

ssm = boto3.client('ssm')
value = ssm.get_parameter(
    Name='/myapp/prod/db/password',
    WithDecryption=True  # decrypts SecureString via KMS
)['Parameter']['Value']

Secrets Manager vs Parameter Store: Side by Side

Criteria	Secrets Manager	Parameter Store
Primary purpose	Secrets with lifecycle	Config + secrets
Automatic rotation	✅ Built-in	❌ (DIY with Lambda + EventBridge)
Cost	$0.40/secret/month	Free (Standard)
Cross-region replication	✅ Native	❌ (manual)
Cross-account	✅ Resource policies	✅ (via RAM / advanced)
Max value size	64 KB	4 KB (Std) / 8 KB (Adv)
Version history	✅ Staging labels	✅ Version numbers
DB integration	✅ RDS/Aurora/Redshift/DocDB	❌
Hierarchical paths	❌ (naming convention only)	✅ Native
KMS encryption	✅ Always	✅ (SecureString only)

Where KMS Fits

KMS (Key Management Service) is the encryption layer underneath both — it doesn't store secrets, it manages the keys that encrypt them.

┌─────────────────────────────────────────────────┐
│  Secrets Manager / Parameter Store (SecureString) │
│         │                                         │
│         │ encrypts secret value using...          │
│         ▼                                         │
│      KMS Key (customer-managed or AWS-managed)    │
└─────────────────────────────────────────────────┘

When You Interact With KMS Directly

Envelope encryption for your own data (encrypt files, EBS, S3 objects)
Key policies controlling who can decrypt (even admins can be excluded)
Cross-account key sharing for shared encrypted resources
Audit — every decrypt operation logged to CloudTrail

KMS vs CloudHSM

	KMS	CloudHSM
Management	Fully managed	You manage the HSM
Tenancy	Multi-tenant (FIPS 140-2 L3 available)	Single-tenant dedicated HSM
Use case	99% of encryption needs	Strict compliance requiring dedicated hardware
Cost	Per key + per request	Per HSM hour (expensive)

Rule: Use KMS unless a specific regulation mandates dedicated single-tenant HSMs.

Access Patterns

For EC2 / ECS / EKS

Use IAM roles (instance profile, task role, IRSA) — never hardcode credentials:

Application → assumes IAM role → role has secretsmanager:GetSecretValue
           → on specific secret ARN only (least privilege)

For EKS Specifically

Two clean patterns:

External Secrets Operator — syncs Secrets Manager / Parameter Store into Kubernetes secrets
Secrets Store CSI Driver — mounts secrets directly into pods as volumes (never stored in etcd)

For Lambda

Use the Parameters and Secrets Lambda Extension — caches secrets locally, reducing API calls and latency:

Lambda → localhost:2773 (extension cache) → Secrets Manager (only on cache miss)

Cost Optimization

The most common mistake: storing everything in Secrets Manager.

Scenario	Wrong (costly)	Right
200 app config values	200 × $0.40 = $80/month in Secrets Manager	Parameter Store Standard = free
5 database passwords (rotating)	Parameter Store + custom rotation Lambda (fragile)	Secrets Manager = $2/month + reliable rotation
Feature flags	Secrets Manager	Parameter Store or AppConfig
TLS private keys	Parameter Store SecureString	Secrets Manager (needs rotation) or ACM

Strategy: Config → Parameter Store (free). Rotating secrets → Secrets Manager (worth the cost). Non-rotating secrets → Parameter Store SecureString.

Rotation Without Secrets Manager

If you must rotate a Parameter Store secret (to save cost), build it yourself:

EventBridge Scheduler (every 30 days)
    │
    ▼
Lambda rotation function:
  1. Generate new credential
  2. Update the target system
  3. Update Parameter Store SecureString
  4. Notify dependents (or they poll)

But honestly — if you need reliable rotation, Secrets Manager's $0.40/month is cheaper than maintaining this yourself. Only DIY when you have many secrets and rotation is simple.

Common Mistakes

Mistake	Problem	Fix
Everything in Secrets Manager	Paying $0.40/month for static config	Config → Parameter Store (free)
Hardcoded secrets in code/env vars	Exposure risk, no rotation	IAM role + Secrets Manager/Parameter Store
Calling GetSecretValue every request	Latency + API cost	Cache (SDK caching or Lambda extension)
No least-privilege on secret access	Any role can read all secrets	Scope IAM to specific secret ARNs
SecureString for rotating DB creds	Manual rotation is fragile	Secrets Manager with auto-rotation
No CloudTrail on secret access	Can't audit who read what	Enable CloudTrail (both services log)
Committing secrets to Git	Permanent exposure	git-secrets pre-commit + rotate immediately

Decision Flowchart

START
  │
  ├── Is it an encryption KEY (not a secret value)?
  │     └── YES → KMS (or CloudHSM if dedicated HSM required)
  │
  ├── Does it need automatic rotation?
  │     └── YES → Secrets Manager
  │
  ├── Is it a database credential (RDS/Aurora/Redshift/DocDB)?
  │     └── YES → Secrets Manager (native integration)
  │
  ├── Does it need cross-region replication?
  │     └── YES → Secrets Manager
  │
  ├── Is it a secret that rarely/never rotates?
  │     └── YES → Parameter Store (SecureString)
  │
  └── Is it plain configuration?
        └── YES → Parameter Store (String, Standard tier — free)

Summary

AWS secrets management comes down to matching the store to the need:

Secrets Manager — rotating secrets, database credentials, cross-region/cross-account. Worth $0.40/secret/month for the rotation and integration.
Parameter Store — configuration and static secrets. Free standard tier makes it the default for anything not rotating.
KMS — the encryption layer under both. Direct use for envelope encryption and key policies.

The guiding principle: Don't pay Secrets Manager prices for Parameter Store problems. Config and static values go in Parameter Store (free). Reserve Secrets Manager for what actually needs rotation, replication, or database integration. And never — ever — hardcode secrets in code or environment variables when an IAM role + managed store does it securely.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS security architecture and cloud infrastructure automation. Connect on LinkedIn.

EKS Production Hardening Guide: Security, Karpenter, Cost, and Upgrades for Real-World Kubernetes

AlpeshKumbhare — Mon, 31 Aug 2026 06:27:45 +0000

Spinning up an EKS cluster takes 20 minutes. Running one in production without incidents takes a lot more. The gap between "it works in the demo" and "it survives Black Friday" is filled with security hardening, intelligent autoscaling, cost discipline, and upgrade strategy.

This guide covers the production-readiness checklist for EKS: the security controls, the Karpenter setup, the cost optimization, and the day-2 operations that keep clusters healthy at scale.

The EKS Shared Responsibility Model

┌─────────────────────────────────────────────────────────────────┐
│  AWS MANAGES (Control Plane)                                     │
│  • Kubernetes API server, etcd, scheduler, controller manager    │
│  • Control plane HA across 3 AZs                                 │
│  • Control plane patching and availability                       │
├─────────────────────────────────────────────────────────────────┤
│  YOU MANAGE (Data Plane + Config)                                │
│  • Worker nodes (or use EKS Auto Mode / Fargate)                 │
│  • Pod security, network policies, RBAC                          │
│  • Secrets management, image scanning                            │
│  • Add-ons, upgrades, cost optimization                          │
└─────────────────────────────────────────────────────────────────┘

Part 1: Security Hardening

1. RBAC and Least Privilege

Default deny → Grant specific permissions per role

├── Developers: read pods/logs in their namespace only
├── CI/CD: deploy to specific namespaces
├── Platform team: cluster-admin (limited members)
└── Applications: ServiceAccount with minimal permissions (IRSA)

IRSA (IAM Roles for Service Accounts): Map Kubernetes ServiceAccounts to IAM roles. Pods get AWS permissions without node-level credentials.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: s3-reader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/s3-read-role

EKS Pod Identity (newer alternative to IRSA): simpler association, no OIDC trust policy management.

2. Network Policies (Default Deny)

By default, all pods can talk to all pods. Lock it down:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then explicitly allow required traffic. Use the VPC CNI network policy support or Cilium for enforcement.

3. Secrets Encryption

Envelope encryption: Encrypt Kubernetes secrets in etcd using KMS
External Secrets Operator: Sync from Secrets Manager / Parameter Store (don't store secrets in etcd at all)
Never commit secrets to Git or bake into images

# EKS cluster with KMS envelope encryption
encryptionConfig:
  - resources: ["secrets"]
    provider:
      keyArn: arn:aws:kms:eu-west-1:123456789:key/xxx

4. Pod Security Standards

Enforce Pod Security Admission (replaces deprecated PodSecurityPolicy):

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

restricted prevents: privileged containers, host namespace access, running as root, privilege escalation.

5. Image Security

ECR image scanning: Scan on push, block deploy on critical CVEs
Image signing: Cosign/Notation for supply chain integrity
Admission control: Kyverno or OPA Gatekeeper to enforce policies (only signed images, no :latest, resource limits required)

6. Control Plane Logging

Enable all control plane log types → CloudWatch:

api | audit | authenticator | controllerManager | scheduler

The audit log is critical for security forensics — it records every API call to the cluster.

Part 2: Karpenter for Intelligent Compute

Karpenter replaced Cluster Autoscaler as the production standard. It provisions right-sized nodes in seconds based on actual pod requirements.

Why Karpenter Over Cluster Autoscaler

Feature	Cluster Autoscaler	Karpenter
Node selection	Fixed node groups	Dynamic, picks optimal instance type
Speed	Minutes	Seconds
Bin packing	Limited	Intelligent consolidation
Spot handling	Basic	Advanced (diversification, interruption)
Instance flexibility	Per node group	Any instance matching constraints

Karpenter NodePool (v1 API)

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["arm64", "amd64"]  # Graviton + x86
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        name: default
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
  limits:
    cpu: 1000

Karpenter Production Best Practices

SQS interruption handling — configure Spot interruption queue so Karpenter drains nodes gracefully (2-min warning)
NodePool isolation — separate NodePools for system vs application workloads
Disruption budgets — limit how many nodes Karpenter can consolidate at once
AMI pinning — pin AMI versions for predictable upgrades (don't auto-update)
Consolidation — enable to bin-pack workloads and reduce node count
Diversification — allow multiple instance types to survive Spot interruptions

Part 3: Cost Optimization

EKS costs can spiral. Here's how to control them:

Compute Cost Levers

Lever	Savings
Karpenter consolidation	30-50% (bin-packing eliminates waste)
Spot for stateless workloads	Up to 90% vs On-Demand
Graviton (arm64) nodes	20-40% better price/performance
Savings Plans for baseline	Up to 72% for steady-state On-Demand
Right-size pod requests	Prevents over-provisioning nodes

The Baseline + Burst Pattern

Baseline (predictable load)  → On-Demand + Savings Plans (cost-committed)
Burst (variable load)        → Spot Instances (cheap, interruptible)
Critical system pods         → On-Demand (never interrupted)

Right-Sizing Pod Requests

resources:
  requests:
    cpu: 250m      # What the pod actually needs (from metrics)
    memory: 512Mi
  limits:
    memory: 512Mi   # Prevent OOM affecting neighbors
    # No CPU limit — let it burst (CPU is compressible)

Key insight: Over-requesting CPU/memory forces Karpenter to provision more/bigger nodes. Use VPA recommendations or metrics to right-size requests.

Eliminate NAT Gateway Costs

Route AWS API traffic through VPC endpoints (Gateway endpoints for S3/DynamoDB are free):

Without VPC endpoints: Pod → NAT Gateway ($0.045/GB) → S3
With VPC endpoints:    Pod → S3 Gateway Endpoint ($0) → S3

Part 4: Networking

VPC CNI Configuration

Prefix delegation — assign /28 prefixes to ENIs (more pods per node)
Custom networking — pods in separate subnets from nodes
Security groups for pods — apply SGs at pod level (not just node)

Ingress and Load Balancing

Option	Use For
AWS Load Balancer Controller	ALB (L7) or NLB (L4) for ingress
Ingress (ALB)	HTTP/HTTPS routing, path-based
Gateway API	Modern replacement for Ingress (more expressive)
VPC Lattice	Cross-cluster service mesh with IAM auth

Service Mesh (When Needed)

Don't add a service mesh by default. Add Istio/Linkerd only when you need:

mTLS between all services
Advanced traffic management (retries, circuit breaking)
Detailed L7 observability

For simpler needs, VPC Lattice or App Mesh may suffice.

Part 5: Observability

Metrics  → CloudWatch Container Insights + Prometheus (Managed Prometheus)
Logs     → Fluent Bit → CloudWatch Logs / OpenSearch
Traces   → ADOT (OpenTelemetry) → X-Ray
Dashboards → Managed Grafana

Essential Metrics to Monitor

Node CPU/memory pressure
Pod restart counts (crash loops)
Pending pods (capacity issues)
Karpenter provisioning latency
Persistent volume usage
API server latency

Part 6: Upgrades

EKS supports Kubernetes versions for ~14 months. Plan upgrades:

Upgrade Strategy

Read the changelog — check for deprecated APIs (use kubent / Pluto to find them)
Upgrade control plane first — AWS handles this (one minor version at a time)
Upgrade add-ons — VPC CNI, CoreDNS, kube-proxy must be compatible
Upgrade nodes — rolling replacement (Karpenter makes this easy — drain + provision new AMI)
Test in non-prod first — always

EKS Auto Mode (Simplify Everything)

EKS Auto Mode (GA) manages compute, scaling, and upgrades automatically:

AWS manages node provisioning, patching, and Karpenter under the hood
You focus on workloads, not infrastructure
Trade-off: less control, slightly higher cost, but dramatically less ops

Consider Auto Mode when: small platform team, want minimal Kubernetes ops overhead.

Part 7: Resilience

Backup and DR

Velero — backup cluster state and persistent volumes
Multi-AZ — spread nodes across 3 AZs (Karpenter handles this)
Pod Disruption Budgets — ensure minimum replicas during disruptions
Cross-region DR — GitOps (ArgoCD) redeploys to DR cluster from Git

Pod Disruption Budget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Ensures at least 2 API pods stay running during node drains/upgrades.

Production Readiness Checklist

SECURITY
☐ RBAC with least privilege
☐ IRSA / Pod Identity (no node-level AWS creds)
☐ Default-deny NetworkPolicy per namespace
☐ Secrets encrypted (KMS envelope or External Secrets)
☐ Pod Security Standards (restricted)
☐ ECR image scanning + admission control
☐ Control plane audit logging enabled

COMPUTE
☐ Karpenter with Spot + On-Demand + Graviton
☐ SQS interruption handling
☐ Consolidation enabled
☐ Right-sized pod requests

COST
☐ VPC endpoints (eliminate NAT charges)
☐ Savings Plans for baseline
☐ Spot for stateless workloads

OBSERVABILITY
☐ Container Insights + Prometheus
☐ Centralized logging (Fluent Bit)
☐ Distributed tracing (ADOT)

RESILIENCE
☐ Multi-AZ node distribution
☐ Pod Disruption Budgets
☐ Velero backups
☐ Tested upgrade path

GITOPS
☐ ArgoCD / Flux for declarative deployments
☐ All cluster config in Git

Common EKS Production Mistakes

Mistake	Impact	Fix
No resource requests/limits	Node overcommit, OOM kills	Set requests based on metrics
All traffic through NAT Gateway	High data processing costs	VPC endpoints for S3/DynamoDB
No NetworkPolicy	Lateral movement if breached	Default-deny + explicit allow
Node-level IAM credentials	Over-privileged pods	IRSA / Pod Identity
Cluster Autoscaler (legacy)	Slow, inefficient scaling	Migrate to Karpenter
No PDB	Outages during upgrades	Pod Disruption Budgets
Ignoring version support window	Forced emergency upgrades	Scheduled upgrade cadence
Secrets in etcd unencrypted	Exposure risk	KMS envelope encryption

Summary

Production EKS comes down to seven pillars:

Security — RBAC, IRSA, NetworkPolicies, Pod Security Standards, image scanning, audit logging
Compute — Karpenter with Spot + On-Demand + Graviton, intelligent consolidation
Cost — VPC endpoints, Savings Plans, Spot, right-sized requests
Networking — VPC CNI tuning, ALB Controller, Gateway API
Observability — Container Insights, Prometheus, ADOT tracing
Upgrades — scheduled cadence, add-on compatibility, or EKS Auto Mode
Resilience — multi-AZ, PDBs, Velero backups, GitOps

If your team is small and Kubernetes ops is a burden, seriously evaluate EKS Auto Mode — it handles most of the compute, scaling, and upgrade complexity so you can focus on workloads.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in Kubernetes, container platforms, and AWS infrastructure automation. Connect on LinkedIn.

Building a Modern Data Lakehouse on AWS: S3, Iceberg, Glue, Athena, and Lake Formation

AlpeshKumbhare — Tue, 25 Aug 2026 11:15:51 +0000

The data lakehouse has become the default architecture for analytics on AWS in 2026. It combines the best of both worlds: the low-cost, schema-flexible storage of a data lake (S3) with the performance, ACID transactions, and governance of a data warehouse — without actually running a warehouse for most workloads.

The enabling technology: Apache Iceberg — an open table format that brings SQL-like capabilities (INSERT, UPDATE, DELETE, time travel) to files sitting in S3. Combined with AWS Glue for ETL, Athena for queries, and Lake Formation for governance, you get a complete analytics platform without managing servers.

Data Lake vs Data Warehouse vs Lakehouse

┌──────────────────┐  ┌──────────────────┐  ┌──────────────────────────┐
│   DATA LAKE      │  │  DATA WAREHOUSE  │  │     DATA LAKEHOUSE       │
│                  │  │                  │  │                          │
│ ✅ Cheap storage │  │ ✅ Fast queries   │  │ ✅ Cheap storage (S3)    │
│ ✅ Schema-on-read│  │ ✅ ACID txns     │  │ ✅ Fast queries (Iceberg)│
│ ✅ Any format    │  │ ✅ Governance    │  │ ✅ ACID transactions     │
│ ❌ No ACID       │  │ ❌ Expensive     │  │ ✅ Governance (LF)       │
│ ❌ No updates    │  │ ❌ Vendor lock-in│  │ ✅ Open format           │
│ ❌ Stale data    │  │ ❌ Schema-rigid  │  │ ✅ Schema evolution      │
│                  │  │                  │  │ ✅ Time travel           │
│  (S3 + Parquet)  │  │  (Redshift)      │  │ (S3 + Iceberg + Athena) │
└──────────────────┘  └──────────────────┘  └──────────────────────────┘

The AWS Lakehouse Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         CONSUMPTION                                   │
│  Athena (SQL) | Redshift Spectrum | EMR (Spark) | QuickSight (BI)   │
├─────────────────────────────────────────────────────────────────────┤
│                         GOVERNANCE                                    │
│  Lake Formation (permissions, audit) | Glue Data Catalog (metadata) │
├─────────────────────────────────────────────────────────────────────┤
│                         TABLE FORMAT                                  │
│  Apache Iceberg (ACID, time travel, schema evolution, compaction)    │
├─────────────────────────────────────────────────────────────────────┤
│                         PROCESSING                                    │
│  Glue ETL (Spark) | Glue Streaming | EMR | Zero-ETL | Firehose     │
├─────────────────────────────────────────────────────────────────────┤
│                         INGESTION                                     │
│  Kinesis | DMS | AppFlow | S3 Transfer | Direct PUT                 │
├─────────────────────────────────────────────────────────────────────┤
│                         STORAGE                                       │
│  Amazon S3 (raw / curated / analytics zones)                         │
└─────────────────────────────────────────────────────────────────────┘

Apache Iceberg: The Game Changer

Iceberg is an open table format that sits between your query engine and S3 files. It enables data warehouse capabilities on data lake storage:

What Iceberg Adds to S3

Capability	Without Iceberg (Parquet on S3)	With Iceberg
INSERT/UPDATE/DELETE	❌ Append-only	✅ Row-level operations
ACID transactions	❌ Partial writes possible	✅ Atomic commits
Schema evolution	❌ Break consumers	✅ Add/rename/drop columns safely
Time travel	❌ No history	✅ Query any past snapshot
Partition evolution	❌ Rewrite all data	✅ Change partitioning without rewrite
Hidden partitioning	❌ User must know partition scheme	✅ Engine handles automatically
Concurrent writes	❌ Last write wins	✅ Optimistic concurrency control

Iceberg on AWS: Service Support

Service	Iceberg Support
Athena	Full read/write (CREATE TABLE, INSERT, UPDATE, DELETE, MERGE)
Glue ETL	Full read/write via Spark connector
EMR	Full support (Spark, Trino, Flink)
Redshift Spectrum	Read access to Iceberg tables
Glue Data Catalog	Native Iceberg catalog integration
Lake Formation	Fine-grained access control on Iceberg tables

Creating an Iceberg Table (Athena)

CREATE TABLE analytics.orders (
    order_id STRING,
    customer_id STRING,
    amount DECIMAL(10,2),
    status STRING,
    order_date TIMESTAMP,
    region STRING
)
PARTITIONED BY (region, month(order_date))
LOCATION 's3://my-lakehouse/analytics/orders/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

Note: month(order_date) is hidden partitioning — queries don't need to know the partition scheme. Iceberg handles pruning automatically.

Row-Level Operations

-- Update order status
UPDATE analytics.orders
SET status = 'shipped'
WHERE order_id = 'ORD-12345';

-- Delete cancelled orders older than 1 year
DELETE FROM analytics.orders
WHERE status = 'cancelled' AND order_date < current_timestamp - interval '1' year;

-- Merge (upsert) from staging
MERGE INTO analytics.orders target
USING staging.new_orders source
ON target.order_id = source.order_id
WHEN MATCHED THEN UPDATE SET status = source.status, amount = source.amount
WHEN NOT MATCHED THEN INSERT (order_id, customer_id, amount, status, order_date, region)
VALUES (source.order_id, source.customer_id, source.amount, source.status, source.order_date, source.region);

Time Travel

-- Query data as it was 24 hours ago
SELECT * FROM analytics.orders
FOR TIMESTAMP AS OF (current_timestamp - interval '24' hour);

-- Query a specific snapshot
SELECT * FROM analytics.orders
FOR VERSION AS OF 12345678901234;

-- View snapshot history
SELECT * FROM "analytics"."orders$snapshots";

Data Lake Zones: Organizing Your S3 Storage

s3://company-lakehouse/
├── raw/                    ← Landing zone (source format, immutable)
│   ├── crm/
│   ├── erp/
│   └── clickstream/
│
├── curated/                ← Cleaned, validated, Iceberg format
│   ├── customers/
│   ├── orders/
│   └── products/
│
└── analytics/              ← Aggregated, business-ready
    ├── daily_revenue/
    ├── customer_360/
    └── product_performance/

Zone	Format	Purpose	Access
Raw	Source format (JSON, CSV, Parquet)	Immutable landing zone, audit trail	Data engineers only
Curated	Iceberg (optimized Parquet)	Cleaned, validated, single source of truth	Data engineers + analysts
Analytics	Iceberg (aggregated)	Business-ready datasets, dashboards	Analysts + BI tools

AWS Glue: The ETL Engine

Glue Components

Component	Purpose
Glue Data Catalog	Centralized metadata (databases, tables, schemas) — the "card catalog"
Glue ETL Jobs	Spark-based data transformation (Python/Scala)
Glue Crawlers	Auto-discover schemas from S3 data, populate catalog
Glue Studio	Visual ETL designer (no-code/low-code)
Glue Streaming	Near-real-time ETL from Kinesis/Kafka/MSK
Glue Data Quality	Define and enforce quality rules on datasets

Glue ETL Job: Raw → Curated (Iceberg)

import sys
from awsglue.transforms import *
from awsglue.context import GlueContext
from pyspark.context import SparkContext

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

# Read raw data
raw_df = spark.read.json("s3://company-lakehouse/raw/orders/2026/08/25/")

# Transform: clean, validate, enrich
curated_df = (raw_df
    .filter(raw_df.order_id.isNotNull())
    .withColumn("order_date", to_timestamp("order_date_str"))
    .withColumn("amount", col("amount").cast("decimal(10,2)"))
    .drop("order_date_str", "_metadata")
)

# Write to Iceberg table (append or merge)
curated_df.writeTo("glue_catalog.curated.orders") \
    .using("iceberg") \
    .append()

Lake Formation: Governance Layer

Lake Formation provides fine-grained access control — column-level and row-level security on your lakehouse:

Permission Model

Lake Formation Permissions:
├── Database-level: Who can see which databases?
├── Table-level: Who can query which tables?
├── Column-level: Who can see which columns? (mask PII)
└── Row-level: Who can see which rows? (data filters by region/team)

Row-Level Security Example

Policy: "EU Data Residency"
├── Principal: EU-Analytics-Team
├── Table: curated.customers
├── Filter: region IN ('eu-west-1', 'eu-central-1')
└── Effect: Team can ONLY see EU customers

Tag-Based Access Control (LF-TBAC)

Instead of managing permissions per table, assign tags and grant access by tag:

Tag: classification = "pii"
  → Applied to: customers.email, customers.phone, orders.billing_address
  → Grant: Only "PII-Authorized" group can see these columns
  → Everyone else: columns masked or hidden

Amazon Athena: Serverless Queries

Athena queries Iceberg tables directly on S3 — no infrastructure to manage:

Serverless — pay per query ($5 per TB scanned)
Federated queries — query RDS, DynamoDB, Redshift alongside S3 in one SQL statement
Prepared statements — parameterized queries for applications
Workgroups — separate teams with cost controls and query limits

Cost Optimization for Athena

Technique	Savings
Columnar format (Parquet/ORC via Iceberg)	30-90% less data scanned
Partition pruning (Iceberg hidden partitioning)	Scan only relevant partitions
Compression (Snappy/ZSTD)	50-70% less storage + scan cost
CTAS for materialized views	Pre-compute expensive joins
Workgroup byte limits	Prevent runaway queries

Zero-ETL Integrations

AWS is pushing "zero-ETL" — direct integration between operational databases and analytics without building ETL pipelines:

Source	Destination	What It Does
Aurora → Redshift	Zero-ETL	Near-real-time replication without Glue jobs
DynamoDB → OpenSearch	Zero-ETL	Automatic sync for search/analytics
DynamoDB → Redshift	Zero-ETL	Export DynamoDB data for analytics
RDS → S3 (via DMS)	CDC	Change data capture for lakehouse ingestion

When Zero-ETL vs Glue ETL

Use Zero-ETL when...	Use Glue ETL when...
Source → destination without transformation	Need data cleansing, validation, enrichment
Supported source/destination pair	Custom transformation logic
Minimal latency requirements	Complex multi-source joins
Simple replication	Business rule application

Real-Time Lakehouse Pattern

For near-real-time analytics, combine streaming ingestion with Iceberg:

App Events → Kinesis Data Streams → Glue Streaming ETL → Iceberg Table
                                                              │
                                              Athena (queries latest data)

Glue Streaming writes micro-batches to Iceberg every 1-5 minutes. Athena queries see near-real-time data without separate real-time infrastructure.

Table Maintenance (Iceberg Housekeeping)

Iceberg tables need periodic maintenance for optimal performance:

Operation	What It Does	Frequency
Compaction	Merge small files into larger ones (better query performance)	Daily
Expire snapshots	Remove old snapshots (reduce metadata overhead + storage)	Weekly
Remove orphan files	Delete files not referenced by any snapshot	Weekly
Rewrite manifests	Optimize manifest file layout	As needed

Athena OPTIMIZE (Compaction)

-- Compact small files for better query performance
OPTIMIZE analytics.orders REWRITE DATA USING BIN_PACK;

-- Expire old snapshots (keep last 7 days)
ALTER TABLE analytics.orders SET TBLPROPERTIES (
    'vacuum_min_snapshots_to_keep' = '10',
    'vacuum_max_snapshot_age_seconds' = '604800'
);
VACUUM analytics.orders;

Common Mistakes

Mistake	Problem	Fix
No partitioning strategy	Full table scans on every query	Use Iceberg hidden partitioning (by date, region)
Too many small files	Slow queries, high S3 API costs	Regular compaction (OPTIMIZE)
No snapshot expiry	Storage costs grow forever	VACUUM old snapshots weekly
Raw zone with no schema validation	Bad data propagates downstream	Glue Data Quality rules at ingestion
Everyone queries raw zone directly	Inconsistent results, no governance	Force queries through curated/analytics zones
Column-level security as afterthought	PII exposed to unauthorized users	Design Lake Formation tags from day one
Redshift for everything	Expensive for exploratory queries	Athena for ad-hoc, Redshift for heavy BI only

Summary

The modern AWS lakehouse in 2026:

Storage: S3 (cheap, durable, infinitely scalable)
Table format: Apache Iceberg (ACID, time travel, schema evolution, hidden partitioning)
Catalog: Glue Data Catalog (centralized metadata for all engines)
Processing: Glue ETL (batch + streaming) or Zero-ETL (for supported pairs)
Query: Athena (serverless SQL, pay per scan) + Redshift Spectrum (for heavy BI)
Governance: Lake Formation (column/row-level security, tag-based access)
BI: QuickSight (serverless dashboards connected to Athena/Redshift)

The key shift: You don't need a data warehouse for most analytics anymore. Iceberg on S3 + Athena gives you warehouse-like capabilities at data lake prices. Add Redshift only when you need sub-second complex aggregations on petabytes.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS data architecture and cloud infrastructure automation. Connect on LinkedIn.

AWS Continuum: AI-Powered Security at Machine Speed — What It Is, How It Works, and Why It Changes AppSec

AlpeshKumbhare — Mon, 24 Aug 2026 06:12:14 +0000

In June 2026, AWS launched Continuum at AWS Summit NYC — an AI-native security platform that handles the full vulnerability lifecycle: discover, prioritize, validate, and remediate. In August 2026, they expanded it with integrations into Anthropic Claude Code and OpenAI Codex, bringing security intelligence directly into AI-assisted coding workflows.

This isn't another scanner. It's a security agent that reasons about your environment, confirms what's actually exploitable, and drives toward resolution — at machine speed, within guardrails you define.

What Is AWS Continuum?

AWS Continuum is an AI-powered security service that manages the full lifecycle of code vulnerabilities:

┌────────────────────────────────────────────────────────────────────┐
│                   AWS CONTINUUM LIFECYCLE                            │
│                                                                      │
│  DISCOVER ──→ PRIORITIZE ──→ VALIDATE ──→ REMEDIATE ──→ VERIFY     │
│                                                                      │
│  Find vulns     Rank by       Confirm       Generate      Confirm   │
│  in code,       exploitability exploitable   fix, create   fix works │
│  deps, infra    + blast radius (not just     PR, deploy   (re-test) │
│                               theoretical)                           │
└────────────────────────────────────────────────────────────────────┘

The key differentiator: Most scanners find vulnerabilities. Continuum validates whether they're actually exploitable in YOUR environment and context, then fixes them automatically.

The Problem Continuum Solves

Traditional AppSec workflow in 2026:

Scanner finds 500 vulnerabilities
  → Security team triages (2 weeks)
    → 80% are false positives or not exploitable
      → 100 actual issues remain
        → Dev team prioritizes (1 week)
          → Fixes deployed over next sprint
            → Total time: 4-6 weeks from detection to fix

Continuum workflow:

Continuum discovers vulnerabilities
  → AI validates exploitability (minutes)
    → Confirmed issues prioritized by blast radius
      → Automated remediation (PR generated)
        → Developer reviews and merges
          → Total time: hours, not weeks

AWS Continuum Components

1. Continuum for Code Vulnerabilities

The core service — continuous security for your application code and dependencies:

Discovery: Scans code repositories, dependencies, and container images
Prioritization: AI ranks by real-world exploitability, not just CVSS score
Validation: Confirms vulnerabilities are reachable in your specific code paths
Remediation: Generates fixes, creates pull requests, monitors deployment
Verification: Re-tests after fix to confirm resolution

2. AWS Security Agent

A frontier AI agent that performs on-demand penetration testing:

Deploys specialized AI agents against your application
Develops application context from documentation and credentials
Executes multi-step attack scenarios (like a human pentester)
Discovers complex vulnerabilities that static scanners miss
Reports findings with full attack chain documentation

3. Continuum + AI Coding Assistants (August 2026)

The latest expansion — security intelligence embedded directly in developer workflows:

Claude Code integration: Anthropic's AI assistant gets Continuum context when writing/reviewing code
OpenAI Codex integration: Same security intelligence in Codex-powered workflows
Kiro integration: AWS's own IDE incorporates Continuum findings

When AI generates code, Continuum validates it against known vulnerability patterns and your organization's security policies — before it reaches production.

How It Works: Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    YOUR ENVIRONMENT                               │
│  Code Repos (GitHub, CodeCommit) | Container Images (ECR)        │
│  Running Applications | Infrastructure (CloudFormation/Terraform) │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│                    AWS CONTINUUM                                  │
│                                                                   │
│  ┌─────────┐  ┌───────────┐  ┌───────────┐  ┌──────────────┐  │
│  │Discovery│→ │Prioritize │→ │ Validate  │→ │  Remediate   │  │
│  │         │  │           │  │           │  │              │  │
│  │• SAST   │  │• AI triage│  │• Confirm  │  │• Generate fix│  │
│  │• SCA    │  │• Context  │  │  exploit- │  │• Create PR   │  │
│  │• Secrets│  │  aware    │  │  ability  │  │• Auto-deploy │  │
│  │• Config │  │• Blast    │  │• Attack   │  │• Re-verify   │  │
│  │         │  │  radius   │  │  path     │  │              │  │
│  └─────────┘  └───────────┘  └───────────┘  └──────────────┘  │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │              GUARDRAILS (You Define)                       │   │
│  │  • What can be auto-fixed vs requires approval            │   │
│  │  • Scope boundaries (which repos, which envs)             │   │
│  │  • Remediation policies (PR review required, staging test)│   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Key Capabilities

Contextual Prioritization (Not Just CVSS)

Traditional scanners rank by CVSS score. A "Critical 9.8" vulnerability in a library function that's never called in your code is actually zero risk. Continuum uses AI to assess:

Reachability: Is the vulnerable code path actually reachable from your application's entry points?
Exploitability: Given your environment (WAF rules, network config, IAM policies), can this be exploited?
Blast radius: If exploited, what's the impact? (data exposure, lateral movement, privilege escalation)
Business context: Is this in a production-critical service or a development tool?

Automated Validation

Continuum doesn't just report — it confirms. The Security Agent:

Analyzes the vulnerability and your application architecture
Develops a multi-step attack scenario
Executes the attack in a controlled manner (like an automated pentester)
Confirms whether the vulnerability is actually exploitable
Documents the full attack chain for the development team

Automated Remediation

For confirmed vulnerabilities, Continuum generates fixes:

Dependency updates: Creates PR updating vulnerable library to patched version
Code fixes: Generates code patches for application-level vulnerabilities
Configuration changes: Proposes infrastructure configuration remediations
Staging validation: Tests the fix in a staging environment before promoting

All within guardrails you define — you control what gets auto-fixed vs what requires human approval.

Integration Points

Developer Workflow Integration

Developer writes code
       │
       ├── In IDE (Claude Code / Codex / Kiro)
       │     └── Continuum provides real-time security feedback
       │
       ├── On Push (GitHub / CodeCommit)
       │     └── Continuum scans changes, comments on PR
       │
       ├── In Pipeline (CodeBuild / GitHub Actions)
       │     └── Continuum gates deployment on security posture
       │
       └── In Production
              └── Continuum continuously monitors, validates, remediates

AWS Service Integration

Integrates With	How
GitHub	PR comments, auto-fix PRs, repository scanning
CodeCommit	Same as GitHub (native AWS)
ECR	Container image vulnerability scanning
CodeBuild	Pipeline gate — block deploy on critical findings
Security Hub	Findings forwarded for unified security dashboard
CloudTrail	Audit trail of all Continuum actions
IAM	Scoped permissions for what Continuum can access/modify
Claude Code / Codex	Real-time security context during AI-assisted coding

Continuum vs Existing Security Tools

Tool	What It Does	Continuum Advantage
CodeGuru Security	SAST scanning	Continuum validates exploitability + auto-remediates
Inspector	CVE scanning for EC2/ECR/Lambda	Continuum adds AI reasoning about real-world risk
GuardDuty	Runtime threat detection	Continuum is proactive (pre-production), GuardDuty is reactive (runtime)
Security Hub	Aggregates findings	Continuum feeds INTO Security Hub with validated findings
Third-party SAST (Snyk, Checkmarx)	Code scanning	Continuum adds validation + auto-remediation + AI coding assistant integration

Continuum doesn't replace these tools — it adds an AI reasoning layer on top. It can consume findings from Inspector, CodeGuru, and third-party scanners, then validate and remediate them.

Setting Up Continuum

Prerequisites

AWS account with Security Hub enabled
Code repository connected (GitHub, CodeCommit, GitLab)
IAM role for Continuum with appropriate permissions
Define guardrails (what can auto-remediate vs requires approval)

Guardrail Configuration

Guardrails:
├── Auto-fix: Dependency version bumps (low risk)
├── Auto-fix + Staging test: Code-level patches
├── Require approval: Infrastructure changes
├── Require approval: Breaking API changes
└── Block: Never auto-fix (cryptographic changes, auth logic)

Scope Configuration

Scope:
├── Repositories: [repo-1, repo-2, repo-3]
├── Environments: [production, staging]
├── Languages: [Python, Java, TypeScript, Go]
└── Exclusions: [test/**, docs/**, vendor/**]

Security Agent: AI Pentesting

The Security Agent component is particularly interesting — it's essentially an AI penetration tester:

How Security Agent Works

Context gathering: Agent reads application documentation, API specs, architecture diagrams
Threat modeling: AI identifies likely attack vectors based on application type
Attack planning: Develops multi-step attack scenarios (SQLi → privilege escalation → data exfil)
Execution: Runs attacks against your application (in controlled manner)
Reporting: Documents successful attack chains with step-by-step reproduction

Use Cases

Pre-release security review (before production deployment)
Quarterly penetration testing (continuous, not annual)
Compliance validation (demonstrate attack testing for auditors)
Red team exercises (automated adversary simulation)

Security Considerations

Agent operates within defined scope boundaries
Credentials are scoped and time-limited
All actions are logged to CloudTrail
Blast radius controlled by IAM permissions
Can be restricted to non-production environments

How Continuum Fits in DevSecOps

┌─────────────────────────────────────────────────────────────────┐
│  DEVELOPMENT                                                     │
│  IDE (Continuum in Claude/Codex) → pre-commit scanning           │
├─────────────────────────────────────────────────────────────────┤
│  BUILD                                                           │
│  Pipeline scan → PR comments → dependency check → container scan │
├─────────────────────────────────────────────────────────────────┤
│  TEST                                                            │
│  Security Agent pentest → staging validation → fix verification  │
├─────────────────────────────────────────────────────────────────┤
│  DEPLOY                                                          │
│  Gate on security posture → no critical unresolved findings      │
├─────────────────────────────────────────────────────────────────┤
│  OPERATE                                                         │
│  Continuous monitoring → new CVEs checked → auto-remediate       │
└─────────────────────────────────────────────────────────────────┘

What Makes Continuum Different

Full lifecycle — not just scanning, but validation, prioritization, remediation, and verification
AI-native — uses frontier models to reason about exploitability, not just pattern matching
Context-aware — understands YOUR code, YOUR infrastructure, YOUR environment
Developer-integrated — embedded in Claude Code, Codex, and Kiro (not a separate tool to check)
Guardrail-controlled — you define what it can auto-fix and what requires human approval
Validates, not just reports — confirms exploitability via Security Agent (AI pentesting)

Summary

AWS Continuum represents a shift from "scan and report" to "discover, validate, and fix":

Discovery: Continuous scanning of code, dependencies, containers, and infrastructure
Prioritization: AI-powered, context-aware ranking (not just CVSS scores)
Validation: Security Agent confirms exploitability via automated attack simulation
Remediation: Generates fixes, creates PRs, validates in staging, all within guardrails
Integration: Embedded in AI coding assistants (Claude Code, Codex) for shift-left security

For teams drowning in scanner noise — hundreds of findings, most not exploitable — Continuum's validation and auto-remediation capabilities represent a significant operational improvement. The integration with AI coding assistants means security feedback arrives while code is being written, not weeks later in a vulnerability report.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS security architecture and cloud infrastructure automation. Connect on LinkedIn.

AWS Serverless Patterns and Anti-Patterns: What Works, What Breaks, and When to Use What

AlpeshKumbhare — Fri, 21 Aug 2026 09:32:42 +0000

Serverless on AWS isn't "just use Lambda." It's a design philosophy: let AWS manage the infrastructure, pay only for what you use, and build with managed services that scale independently. But the patterns that work in serverless are fundamentally different from traditional architectures — and the anti-patterns are expensive to learn the hard way.

This guide covers the patterns that work in production, the anti-patterns that waste money or cause outages, and the decision framework for when serverless is the right (or wrong) choice.

The Serverless Building Blocks

┌─────────────────────────────────────────────────────────────────────┐
│                    AWS SERVERLESS STACK                               │
├─────────────────────────────────────────────────────────────────────┤
│  COMPUTE        │ Lambda | Fargate (serverless containers)           │
│  API            │ API Gateway (REST/HTTP/WebSocket) | AppSync (GraphQL)│
│  ORCHESTRATION  │ Step Functions | EventBridge Scheduler             │
│  MESSAGING      │ SQS | SNS | EventBridge                          │
│  STORAGE        │ S3 | DynamoDB | Aurora Serverless                  │
│  STREAMING      │ Kinesis | DynamoDB Streams | MSK Serverless        │
│  AUTH           │ Cognito | IAM | Lambda Authorizers                 │
│  OBSERVABILITY  │ CloudWatch | X-Ray | Application Signals           │
└─────────────────────────────────────────────────────────────────────┘

Key principle: In serverless, you compose applications from managed services. Lambda is the glue between them — not the application itself.

Pattern 1: Synchronous API (Request/Response)

The most common serverless pattern: HTTP API backed by Lambda.

Client → API Gateway → Lambda → DynamoDB / Aurora Serverless
                                    │
                              Response ← ─ ─ ─ ─ ─ ─ ┘

Best Practices

API Gateway HTTP API (not REST API) — cheaper, faster, simpler for most cases
One Lambda per route (single responsibility) — not a monolith Lambda
Keep Lambda warm — use Provisioned Concurrency for latency-sensitive endpoints
DynamoDB for simple access patterns — scales with traffic, no connection pooling
Aurora Serverless v2 for complex queries — but use RDS Proxy to manage connections

When to Choose HTTP API vs REST API

Feature	HTTP API	REST API
Cost	$1.00/million requests	$3.50/million requests
Latency	Lower (~10ms added)	Higher (~30ms added)
Features	JWT auth, CORS, Lambda integration	WAF, usage plans, API keys, caching, request validation
Choose when	Standard APIs, cost-sensitive	Need WAF, throttling plans, request transforms

Pattern 2: Async Event Processing

Events trigger Lambda. Processing happens independently of the caller.

S3 Upload ──→ Lambda: process image ──→ S3: store thumbnail
SQS Message ──→ Lambda: process order ──→ DynamoDB: update status
EventBridge ──→ Lambda: handle event ──→ SNS: send notification

Best Practices

Always use Dead Letter Queues (DLQ) — failed events go to DLQ, not lost
Design for idempotency — events may be delivered more than once
Batch processing — SQS Lambda trigger processes up to 10 messages per invocation (cost efficient)
Set reserved concurrency — prevent one function from consuming all account concurrency
Use event filtering — Lambda event source filtering reduces invocations (cheaper + simpler)

Event Source Filtering Example

{
  "FilterCriteria": {
    "Filters": [
      {
        "Pattern": "{\"body\": {\"status\": [\"critical\"]}}"
      }
    ]
  }
}

Lambda only invokes for messages where body.status == "critical". Other messages are filtered out at the service level (free).

Pattern 3: Workflow Orchestration (Step Functions)

For multi-step processes with branching, retries, and error handling.

Step Function:
├── Validate input
├── Process payment (Lambda)
│     ├── Success → Reserve inventory (Lambda)
│     └── Failure → Notify customer (SNS) → End
├── Ship order (Lambda)
├── Wait 7 days
└── Send follow-up email (Lambda)

Step Functions: Express vs Standard

Feature	Standard	Express
Duration	Up to 1 year	Up to 5 minutes
Pricing	Per state transition ($0.025/1000)	Per execution + duration
Execution model	Exactly-once	At-least-once
History	Full execution history (90 days)	CloudWatch Logs only
Use case	Long-running workflows, human approval	High-volume, short processing (ETL, transforms)

Direct Service Integrations (Skip Lambda)

Step Functions can call 200+ AWS services directly without Lambda:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::dynamodb:putItem",
  "Parameters": {
    "TableName": "Orders",
    "Item": {
      "orderId": {"S.$": "$.orderId"},
      "status": {"S": "confirmed"}
    }
  }
}

No Lambda needed — Step Functions writes to DynamoDB directly. Cheaper, fewer moving parts, lower latency.

Rule: If your Lambda only calls one AWS API — replace it with a direct integration.

Pattern 4: Streaming / Real-Time Processing

For continuous data ingestion and processing.

IoT Devices ──→ Kinesis ──→ Lambda (real-time) ──→ DynamoDB
                  │
                  └──→ Firehose ──→ S3 (data lake)

Best Practices

Kinesis for ordering and replay — Lambda for real-time processing
Firehose for batched delivery — no code needed for S3/Redshift/OpenSearch
Tumbling windows — Lambda aggregates over time windows natively
Bisect on error — Kinesis + Lambda can split failed batches to isolate the bad record

Pattern 5: Fan-Out / Scatter-Gather

One trigger spawns many parallel processes, results are aggregated.

API → Step Function (Distributed Map):
  ├── Process item 1 (Lambda)
  ├── Process item 2 (Lambda)
  ├── Process item 3 (Lambda)
  └── ... (10,000 concurrent)

  → Aggregate results → Response

Step Functions Distributed Map processes millions of items with up to 10,000 concurrent executions. Use for:

Batch processing large datasets from S3
Parallel API calls to external services
Large-scale data transformation

Pattern 6: GraphQL API (AppSync)

For applications needing flexible, client-driven queries.

Client → AppSync → Resolvers:
  ├── DynamoDB (direct resolver, no Lambda)
  ├── Lambda (complex logic)
  ├── Aurora (SQL queries)
  └── HTTP (external APIs)

AppSync advantages over API Gateway + Lambda:

Client fetches exactly what it needs (no over-fetching)
Real-time subscriptions (WebSocket) built in
Direct DynamoDB/Aurora resolvers (no Lambda needed for CRUD)
Caching built in

Pattern 7: Scheduled Tasks

Replace cron servers with serverless scheduling.

EventBridge Scheduler → Lambda: run cleanup
EventBridge Rule (rate/cron) → Lambda: generate report
Step Functions Wait → Lambda: send reminder

EventBridge Scheduler vs EventBridge Rules

Feature	Scheduler	Rules
One-time events	✅ (at specific time)	❌
Timezone support	✅ (handles DST)	❌ (UTC only)
Scale	Millions of schedules	Limited rules per bus
Use case	Per-entity schedules (user reminders)	System-wide recurring jobs

Anti-Patterns: What NOT to Do

Anti-Pattern 1: Lambda Monolith

The mistake: Putting your entire Express/Flask app inside one Lambda function.

❌ BAD: Single Lambda handles ALL routes
   /users, /orders, /products, /admin → one 50MB Lambda

✅ GOOD: One Lambda per route (or per domain)
   /users → users-handler
   /orders → orders-handler

Why it fails: Cold starts scale with package size. One change requires redeploying everything. No independent scaling per endpoint.

Anti-Pattern 2: Lambda Calling Lambda (Synchronous Chain)

The mistake: Lambda A calls Lambda B which calls Lambda C, all synchronously.

❌ BAD:
   Lambda A → invoke → Lambda B → invoke → Lambda C
   (paying for A's time while waiting for B and C)

✅ GOOD:
   Step Functions: A → B → C (orchestrated, not nested)
   Or: A → SQS → B → SQS → C (async, decoupled)

Why it fails: You pay for idle time while waiting. Retry logic becomes complex. Timeouts cascade. Use Step Functions or async messaging instead.

Anti-Pattern 3: Recursive Lambda

The mistake: Lambda invokes itself (or triggers a loop).

❌ DANGEROUS:
   Lambda → writes to S3 → triggers same Lambda → writes to S3 → ...
   (infinite loop = infinite bill)

Fix: Use separate buckets for input/output, or use event source filtering to exclude your own writes.

Anti-Pattern 4: VPC Lambda Without Need

The mistake: Putting Lambda in a VPC "for security" when it doesn't access VPC resources.

Why it fails: VPC Lambda has cold start overhead (ENI creation). If Lambda only calls DynamoDB, S3, or external APIs — it doesn't need VPC. Use VPC only when accessing RDS, ElastiCache, or private EC2 services.

Anti-Pattern 5: Over-Orchestration

The mistake: Using Step Functions for a simple sequential call that could be a direct Lambda + SDK call.

❌ OVER-ENGINEERED:
   Step Function → Lambda (validate) → Lambda (save to DDB)
   (3 resources for what one Lambda could do)

✅ APPROPRIATE:
   Lambda: validate + save to DDB (if it's simple sequential logic)

Rule: Use Step Functions when you need branching, retries, parallel execution, wait states, or error handling across multiple services. Don't use it for simple A→B flows.

Anti-Pattern 6: Ignoring Cold Starts in Latency-Sensitive Paths

The mistake: Using Lambda for a user-facing API with p99 latency SLA of <100ms without addressing cold starts.

Cold start impact:

Python/Node.js: 100-300ms
Java/.NET: 500-3000ms
VPC Lambda: adds 200-500ms on top

Fix:

Provisioned Concurrency (pre-warm instances)
Lambda SnapStart (Java only — restore from checkpoint)
Keep Lambda small (fewer dependencies = faster cold start)
Or use Fargate for latency-critical paths with predictable traffic

Anti-Pattern 7: No Concurrency Limits

The mistake: No reserved concurrency on Lambdas in a shared account.

What happens: One function handling a burst consumes all 1,000 concurrent executions → other functions throttled → cascading failures.

Fix: Set reserved concurrency on critical functions. Set account-level concurrency limits on non-critical batch functions.

When Serverless Fits vs When It Doesn't

Serverless Wins

Characteristic	Why
Variable/unpredictable traffic	Scales to zero, no idle cost
Event-driven workloads	Natural fit for trigger-based processing
Rapid prototyping	Ship in hours, not days
Per-request cost model preferred	Pay only for what you use
Team wants zero infrastructure ops	No patching, scaling, or capacity planning

Serverless Doesn't Fit

Characteristic	Why	Alternative
Consistent high traffic (always on)	Fargate/EC2 is cheaper at high utilization	Fargate
Execution > 15 minutes	Lambda timeout limit	ECS tasks / Step Functions
Persistent connections (WebSocket server, gRPC stream)	Lambda is request/response	Fargate / EC2
Cold start unacceptable (< 50ms p99)	Lambda can't guarantee this	Fargate / EC2 with ALB
GPU / specialized hardware	Lambda doesn't support GPU	EC2
Large deployment package (> 250MB)	Lambda size limits	Containers on Fargate

Cost Optimization Patterns

Pattern	Savings
ARM (Graviton) Lambda	20% cheaper, often 10-30% faster
Increase memory (reduce duration)	Often cheaper: 256MB × 400ms costs same as 512MB × 180ms
Batch SQS messages (10 per invocation)	10x fewer invocations
Direct integrations (skip Lambda)	No Lambda cost for simple pass-through
Event filtering	Reduce unnecessary invocations
Provisioned Concurrency (only for latency needs)	⚠️ Adds cost — use only where needed

Summary

Serverless on AWS works when you follow these principles:

Compose from managed services — Lambda is glue, not the application
Single responsibility — one function per task, not monolith Lambdas
Async by default — use SQS/EventBridge between services, not synchronous chains
Step Functions for orchestration — don't build state machines in Lambda code
Direct integrations — if Lambda just calls one AWS API, remove it
Design for failure — DLQs everywhere, idempotent handlers, circuit breakers
Know when it doesn't fit — persistent connections, GPU, >15 min, cold-start-sensitive paths → use Fargate

The best serverless architectures look nothing like traditional three-tier apps shrunk into Lambda. They're composed of events, queues, state machines, and managed services with Lambda connecting the dots.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS serverless architecture and cloud infrastructure automation. Connect on LinkedIn.

AWS Networking Decision Guide: Transit Gateway vs VPC Peering vs PrivateLink vs VPC Lattice vs Cloud WAN

AlpeshKumbhare — Wed, 19 Aug 2026 06:08:06 +0000

AWS networking has five ways to connect VPCs and services. They're not interchangeable — each solves a different connectivity problem at a different cost point. Using Transit Gateway where VPC Peering suffices wastes money. Using VPC Peering where Transit Gateway is needed creates an unmanageable mesh.

This guide maps each connectivity option to its sweet spot, compares cost and complexity, and provides architecture patterns for real-world multi-account, multi-region deployments.

The Five Connectivity Options

┌──────────────────────────────────────────────────────────────────────┐
│                    AWS VPC CONNECTIVITY                                │
├──────────────┬──────────────┬───────────────┬──────────┬─────────────┤
│ VPC Peering  │Transit Gateway│  PrivateLink  │VPC Lattice│  Cloud WAN  │
│              │              │               │          │             │
│ Point-to-    │ Hub-and-     │ Service       │ Service  │ Global      │
│ point        │ spoke        │ endpoint      │ mesh     │ network     │
│              │              │               │          │             │
│ 2 VPCs       │ Many VPCs    │ Expose 1      │ Service- │ Multi-region│
│ directly     │ via hub      │ service       │ to-svc   │ backbone    │
│              │              │ privately     │ (L7+IAM) │             │
└──────────────┴──────────────┴───────────────┴──────────┴─────────────┘

Quick Decision Matrix

If you need to...	Use...
Connect 2-3 VPCs directly (simple)	VPC Peering
Connect 5+ VPCs through a central hub	Transit Gateway
Expose a service to other VPCs without network joining	PrivateLink
Service-to-service auth (IAM) across VPCs/accounts	VPC Lattice
Global multi-region network with policy	Cloud WAN
Connect on-premises to AWS	Transit Gateway + VPN/Direct Connect

VPC Peering: Direct Point-to-Point

What it is: A direct, private network link between two VPCs. Traffic uses AWS backbone — no internet, no gateway devices.

Characteristics

Property	Detail
Topology	Point-to-point (1 peering = 2 VPCs)
Transitive	❌ No (A↔B and B↔C doesn't give A↔C)
Cross-account	✅ Yes
Cross-region	✅ Yes (inter-region peering)
Bandwidth	No limit (same as within VPC)
Latency	Lowest (no intermediate hop)
Cost	Free (same-AZ), $0.01/GB (cross-AZ/region)

When to Use VPC Peering

2-3 VPCs that need full network connectivity
Lowest latency requirement (no intermediate router)
Cost-sensitive with high data transfer (no per-hour charge)
Simple topology that won't grow beyond ~5 connections

When NOT to Use VPC Peering

5+ VPCs (creates N×(N-1)/2 peering connections — unmanageable mesh)
Need transitive routing (VPC A talking to VPC C through VPC B)
Need centralized inspection/firewall
Need on-premises connectivity through a single point

Architecture: VPC Peering Mesh

     VPC-A ←──── peer ────→ VPC-B
       ↑                       ↑
       │                       │
     peer                    peer
       │                       │
       ↓                       ↓
     VPC-C ←──── peer ────→ VPC-D

4 VPCs = 6 peering connections
10 VPCs = 45 peering connections ← don't do this

Transit Gateway: The Hub Router

What it is: A regional network hub that connects VPCs, VPN connections, and Direct Connect gateways through a single gateway. Think of it as a cloud router.

Characteristics

Property	Detail
Topology	Hub-and-spoke (star)
Transitive	✅ Yes (all attachments can reach each other via route tables)
Cross-account	✅ Yes (share via RAM)
Cross-region	✅ Yes (TGW peering between regions)
Bandwidth	50 Gbps per VPC attachment
Max attachments	5,000 per TGW
Route tables	Multiple (enables segmentation)
Cost	$0.05/hr per attachment + $0.02/GB processed

When to Use Transit Gateway

5+ VPCs needing connectivity
Hub-and-spoke topology (centralized networking)
On-premises connectivity (VPN or Direct Connect terminates at TGW)
Network segmentation via multiple route tables
Centralized egress (inspection VPC with Network Firewall)
Multi-account landing zone networking

Transit Gateway Architecture: Segmented Network

                    ┌─────────────────────────┐
                    │     Transit Gateway      │
                    │                         │
                    │  Route Table: Prod      │
                    │  Route Table: Non-Prod  │
                    │  Route Table: Shared    │
                    └────┬───────┬───────┬────┘
                         │       │       │
              ┌──────────┤       │       ├──────────┐
              │          │       │       │          │
              ▼          ▼       ▼       ▼          ▼
         ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
         │Prod VPC│ │Dev VPC │ │Shared  │ │Inspect │
         │        │ │        │ │Services│ │VPC     │
         └────────┘ └────────┘ └────────┘ └────────┘

Segmentation: Prod route table only has routes to Shared Services and Inspection VPC. Dev cannot reach Prod directly.

TGW Route Table Segmentation

Route Table	Can Reach	Cannot Reach
Production	Shared Services, Inspection, On-Premises	Development, Sandbox
Development	Shared Services, Inspection	Production
Shared Services	Production, Development	Direct internet
On-Premises	Production, Shared Services	Development

PrivateLink: Service Exposure Without Network Joining

What it is: Exposes a specific service (behind NLB) to other VPCs via an interface endpoint. Consumer VPC gets an ENI in their subnet that routes to the provider's service. Networks don't join — only that one service is accessible.

Characteristics

Property	Detail
Topology	Provider → Consumer (one-directional)
Network joining	❌ No (only specific service is exposed)
Cross-account	✅ Yes
Cross-region	✅ Yes (since 2023)
Protocol	TCP (NLB-based)
Cost	$0.01/hr per endpoint + $0.01/GB processed
Security	Consumer only sees the endpoint ENI, not provider's network

When to Use PrivateLink

Expose ONE service to other VPCs without sharing entire network
SaaS provider offering private connectivity to customers
Shared services (API, database proxy) accessible across accounts
Third-party vendor integration (many AWS Marketplace products use PrivateLink)
Security: consumer cannot scan/discover other resources in provider VPC

Architecture: Shared Service via PrivateLink

Provider Account                    Consumer Account
┌──────────────────┐               ┌──────────────────┐
│  Service (EC2/ECS)│               │                  │
│       ↓          │               │   App Server     │
│     [NLB]        │               │       ↓          │
│       ↓          │               │  VPC Endpoint    │
│  Endpoint Service │◄─────────────│  (ENI in subnet) │
└──────────────────┘  PrivateLink  └──────────────────┘

Consumer's app calls the endpoint ENI's DNS → traffic flows privately to provider's NLB → reaches service. No internet, no peering, no TGW needed.

VPC Lattice: Service-to-Service (L7 + IAM)

What it is: Application-layer service mesh that provides service discovery, traffic management, and IAM-based authentication between services — across VPCs and accounts.

Characteristics

Property	Detail
Layer	L7 (HTTP/HTTPS/gRPC) + TCP
Auth model	IAM (SigV4) — identity-based, not network-based
Cross-account	✅ Native (service networks)
Cross-VPC	✅ No peering/TGW needed
Service discovery	Built-in (DNS-based)
Traffic management	Weighted routing, health checks
Observability	Built-in access logs
Cost	Per request + per GB

When to Use VPC Lattice

Service-to-service communication with IAM auth (Zero Trust)
Cross-account service discovery without network plumbing
Weighted traffic routing between service versions (canary)
Don't want to manage TGW/peering for service connectivity
Need L7 visibility (HTTP method, path, status code in access logs)

VPC Lattice vs PrivateLink

Criteria	PrivateLink	VPC Lattice
Layer	L4 (TCP via NLB)	L7 (HTTP/gRPC + TCP)
Auth	Network-level only	IAM (SigV4) per request
Discovery	Manual DNS config	Built-in service discovery
Traffic mgmt	None	Weighted routing
Direction	One-way (provider → consumer)	Bidirectional
Best for	Exposing services to external consumers	Internal service mesh

VPC Lattice vs Transit Gateway

Criteria	Transit Gateway	VPC Lattice
Layer	L3/L4 (network routing)	L7 (application routing)
Joins networks?	Yes (full network reachability)	No (only specific services)
Auth	Security groups / NACLs	IAM per request
Use case	Full network connectivity	Service-to-service only
Cost model	Per attachment-hour + GB	Per request + GB

Rule: Use TGW when you need network-level connectivity (SSH, database ports, broad access). Use VPC Lattice when you need application-level service communication with identity auth.

Cloud WAN: Global Network Fabric

What it is: A managed global network that connects VPCs across regions and on-premises sites with centralized policy. Think of it as a multi-region Transit Gateway with global policy control.

Characteristics

Property	Detail
Scope	Multi-region, global
Policy	Centralized network policy (segments, sharing, inspection)
On-premises	Direct Connect, Site-to-Site VPN
Segmentation	Network segments (like TGW route tables, but global)
Cost	Core network edge per hour + per GB
Best for	Large enterprises with 10+ regions and complex segmentation

When to Use Cloud WAN vs Transit Gateway

Criteria	Transit Gateway	Cloud WAN
Scope	Single region	Global (multi-region)
Cross-region	TGW peering (manual per region pair)	Automatic (policy-driven)
Policy	Route tables per TGW	Global segment policy
Complexity	Medium	High (enterprise-grade)
Cost	Lower	Higher
Best for	Single-region or 2-3 regions	5+ regions with consistent policy

Cost Comparison

Service	Hourly Cost	Data Processing
VPC Peering	Free	$0.01/GB (cross-AZ/region)
Transit Gateway	$0.05/hr per attachment	$0.02/GB
PrivateLink	$0.01/hr per endpoint	$0.01/GB
VPC Lattice	No hourly	$0.025/GB + $0.10/million requests
Cloud WAN	$0.05/hr per attachment + core edge	$0.02/GB

Cost impact at scale:

10 VPCs connected via TGW: $0.05 × 10 attachments × 730 hours = $365/month (before data transfer)

Same 10 VPCs via VPC Peering: 45 peering connections × $0/month (peering is free, but management overhead is high)

Architecture Patterns

Pattern 1: Multi-Account Landing Zone (Most Common)

Transit Gateway (centralized)
├── Shared Services VPC (AD, CI/CD, DNS)
├── Inspection VPC (Network Firewall, egress)
├── Production VPC (workloads)
├── Staging VPC (workloads)
├── Development VPC (workloads)
└── On-Premises (via Direct Connect / VPN)

+ PrivateLink for specific shared services (database proxy, API gateway)
+ VPC Lattice for microservice-to-microservice auth

Pattern 2: Hybrid Cloud (On-Premises + AWS)

On-Premises Data Center
      │
   Direct Connect (or VPN)
      │
      ▼
Transit Gateway
├── VPC: Production workloads
├── VPC: DR (disaster recovery)
└── VPC: Shared Services (DNS resolver, AD connector)

Pattern 3: SaaS Multi-Tenant (Provider)

SaaS Provider VPC
├── Service behind NLB
└── PrivateLink Endpoint Service
      │
      ├── Customer A VPC (endpoint) 
      ├── Customer B VPC (endpoint)
      └── Customer C VPC (endpoint)

Each customer has private access. Networks never join.

Common Networking Mistakes

Mistake	Problem	Fix
VPC Peering mesh with 10+ VPCs	Unmanageable, no transitive routing	Transit Gateway
TGW for 2-3 VPCs	Over-engineered, unnecessary cost	VPC Peering
All traffic through TGW (including AWS API calls)	High data processing charges	VPC endpoints for S3, DynamoDB (free gateway endpoints)
No network segmentation on TGW	Dev can reach prod	Multiple TGW route tables
PrivateLink when you need bidirectional	PrivateLink is one-way	VPC Lattice or TGW
Overlapping CIDRs between VPCs	Can't peer or TGW connect	Plan CIDR with IPAM from day one
Single AZ for NAT Gateway	AZ failure = no egress	NAT GW per AZ

Summary

AWS VPC connectivity comes down to what you're connecting and why:

Need	Service	Cost
2-3 VPCs, full connectivity, lowest cost	VPC Peering	Free (hourly)
5+ VPCs, hub-and-spoke, centralized routing	Transit Gateway	$$
Expose one service privately, no network joining	PrivateLink	$
Service-to-service with IAM auth, no network plumbing	VPC Lattice	$$
Global multi-region with policy-driven segmentation	Cloud WAN	$$$
On-premises to AWS	TGW + Direct Connect/VPN	$$-$$$

The 2026 pattern for most enterprises: Transit Gateway for network backbone + VPC Lattice for service mesh + PrivateLink for external/vendor services + VPC Peering for the occasional simple 2-VPC connection.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS networking and cloud infrastructure automation. Connect on LinkedIn.

CI/CD on AWS: Deployment Strategies, Pipeline Architectures, and the CodePipeline vs GitHub Actions Decision

AlpeshKumbhare — Tue, 18 Aug 2026 06:03:55 +0000

Shipping code to production should be boring. If your deployments are stressful, your pipeline is wrong. A well-designed CI/CD pipeline makes releases routine, reversible, and observable — whether you deploy once a day or fifty times.

AWS provides both native CI/CD services (CodePipeline, CodeBuild, CodeDeploy) and deep integration with third-party tools (GitHub Actions, GitLab CI). The choice depends on your team's workflow preferences and how much you want to stay inside the AWS ecosystem.

This guide covers the deployment strategies that matter, the pipeline architectures that work, and the decision framework for choosing your CI/CD stack.

Deployment Strategies: How Code Reaches Production

Strategy 1: Rolling Deployment

How it works: Replace instances/tasks gradually. Old and new versions run simultaneously during the transition.

Time 0:  [v1] [v1] [v1] [v1]  (4 tasks, all v1)
Time 1:  [v2] [v1] [v1] [v1]  (1 updated)
Time 2:  [v2] [v2] [v1] [v1]  (2 updated)
Time 3:  [v2] [v2] [v2] [v1]  (3 updated)
Time 4:  [v2] [v2] [v2] [v2]  (complete)

Pros: Simple, no extra infrastructure, works everywhere.
Cons: Two versions run simultaneously (must be backward-compatible). Rollback means rolling forward to v1 again.

AWS implementation: ECS rolling update (default), EKS rolling update, EC2 Auto Scaling group instance refresh.

Strategy 2: Blue/Green Deployment

How it works: Deploy new version (green) alongside old version (blue). Switch traffic atomically. Keep blue alive for instant rollback.

┌─────────────────┐         ┌─────────────────┐
│   BLUE (v1)     │         │   GREEN (v2)    │
│   (serving)     │         │   (staged)      │
└────────┬────────┘         └────────┬────────┘
         │                           │
         ▼                           ▼
┌─────────────────────────────────────────────┐
│              ALB / Route53                    │
│         100% → Blue (until cutover)          │
│         Then: 100% → Green                   │
└─────────────────────────────────────────────┘

Pros: Zero downtime. Instant rollback (switch traffic back to blue). Full testing on green before cutover.
Cons: 2x infrastructure during deployment (cost). Database schema changes need careful handling.

AWS implementation:

ECS native blue/green (GA July 2025) — built into ECS service, no CodeDeploy needed
CodeDeploy blue/green for ECS (original approach)
Route53 weighted routing (for broader blue/green)
ALB target group swap

Strategy 3: Canary Deployment

How it works: Route a small percentage of traffic (1-10%) to the new version. Monitor errors. If healthy, gradually shift more traffic. If unhealthy, route all traffic back to old version.

Step 1:  95% → v1,  5% → v2  (canary)
Step 2:  70% → v1, 30% → v2  (expanding)
Step 3:  50% → v1, 50% → v2  (halfway)
Step 4:   0% → v1, 100% → v2 (complete)

Pros: Minimal blast radius. Real production traffic validates new version. Automatic rollback on alarm.
Cons: Complex routing. Must handle session affinity. Slower than blue/green.

AWS implementation:

ECS native canary/linear (GA October 2025) — percentage-based traffic shifting built into ECS
CodeDeploy canary (Lambda, ECS)
App Mesh / VPC Lattice (weighted routing between versions)
ALB weighted target groups

Strategy 4: Feature Flags (Decouple Deploy from Release)

How it works: Deploy code with new features disabled. Enable features independently via configuration (not deployment). Rollback = toggle flag off.

AWS implementation: AppConfig feature flags (native), LaunchDarkly, or custom DynamoDB-backed flags.

Benefit: Deploy anytime. Release (enable feature) separately. Different features for different users (A/B testing).

ECS Native Deployment Capabilities (2025-2026)

ECS received major deployment upgrades — eliminating the need for CodeDeploy in most container scenarios:

Feature	Release	What It Does
Native blue/green	July 2025	Built-in blue/green without CodeDeploy
Linear/canary	October 2025	Percentage-based traffic shifting
NLB support for linear/canary	February 2026	Canary for TCP/gRPC workloads
Pause/continue	May 2026	Pause deployment for manual validation
Configurable circuit breaker	July 2026	Custom failure thresholds for auto-rollback

ECS Deployment Configuration

{
  "deploymentConfiguration": {
    "deploymentType": "BLUE_GREEN",
    "blueGreenDeploymentConfiguration": {
      "trafficRoutingConfig": {
        "type": "CANARY",
        "canaryConfig": {
          "interval": 300,
          "percentage": 10
        }
      },
      "terminationWaitTimeInMinutes": 60,
      "deploymentCircuitBreaker": {
        "enable": true,
        "rollback": true,
        "failureThreshold": 5
      }
    }
  }
}

This deploys with 10% canary, waits 5 minutes, then shifts remaining traffic. If 5+ tasks fail health checks, automatic rollback triggers.

Pipeline Architecture: AWS-Native Stack

AWS CodePipeline + CodeBuild + CodeDeploy

┌──────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐
│  Source   │────→│   Build   │────→│   Test    │────→│  Deploy   │
│(CodeCommit│     │(CodeBuild)│     │(CodeBuild)│     │(CodeDeploy│
│ or GitHub)│     │           │     │           │     │ or ECS)   │
└──────────┘     └───────────┘     └───────────┘     └───────────┘

CodePipeline: Orchestrates the pipeline stages. Triggers on source change.
CodeBuild: Runs build commands, tests, security scans in managed containers.
CodeDeploy: Handles deployment strategies (rolling, blue/green, canary).

Pipeline Stages Best Practice

Source → Build → Unit Test → SAST Scan → Container Scan → 
  Deploy Dev → Integration Test → Deploy Staging → 
    Load Test → Manual Approval → Deploy Production

Multi-Account Pipeline Pattern

Tooling Account (pipeline lives here)
  │
  ├── Deploy → Dev Account (automatic)
  ├── Deploy → Staging Account (automatic + integration tests)
  └── Deploy → Production Account (manual approval gate)

Cross-account deployment uses IAM roles. Pipeline in tooling account assumes role in target account to deploy.

Pipeline Architecture: GitHub Actions

For teams using GitHub as source control, GitHub Actions provides a complete CI/CD solution with AWS integration:

name: Deploy to ECS
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write  # OIDC for AWS auth
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/github-actions-deploy
          aws-region: eu-west-1

      - uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push image
        run: |
          docker build -t $ECR_REGISTRY/my-app:$GITHUB_SHA .
          docker push $ECR_REGISTRY/my-app:$GITHUB_SHA

      - uses: aws-actions/amazon-ecs-deploy-task-definition@v2
        with:
          task-definition: task-def.json
          service: my-service
          cluster: production
          wait-for-service-stability: true

GitHub Actions + OIDC (No Long-Lived Credentials)

Use OpenID Connect (OIDC) to authenticate GitHub Actions to AWS — no access keys needed:

Create IAM Identity Provider for GitHub in AWS
Create IAM role with trust policy for specific repo/branch
GitHub Actions exchanges OIDC token for temporary AWS credentials

Security: Credentials are short-lived (1 hour), scoped to specific repos, and never stored as secrets.

CodePipeline vs GitHub Actions vs GitLab CI

Criteria	CodePipeline	GitHub Actions	GitLab CI
Source integration	CodeCommit, GitHub, S3, ECR	GitHub (native)	GitLab (native)
Build	CodeBuild (managed)	Hosted runners or self-hosted	Shared or self-hosted runners
AWS integration	Deep (native IAM, VPC, cross-account)	Good (via aws-actions, OIDC)	Good (via CLI, OIDC)
Deployment strategies	CodeDeploy (full support)	Manual (scripts + AWS CLI)	Manual (scripts + AWS CLI)
Pricing	Free pipeline + CodeBuild minutes ($0.005/min)	2000 free min/month, then $0.008/min	400 free min/month, then $0.005/min
Ecosystem	AWS-only	20K+ marketplace actions	500+ templates
Multi-cloud	❌	✅	✅
Pipeline as code	YAML or console	YAML (`.github/workflows/`)	YAML (`.gitlab-ci.yml`)
Approval gates	Manual approval action	Environment protection rules	Manual jobs
Best for	AWS-native teams, complex deployment strategies	Teams on GitHub, multi-cloud	Teams on GitLab, self-hosted preference

When to Choose What

CodePipeline: You want native blue/green/canary via CodeDeploy, cross-account deployment patterns, or deep AWS integration without custom scripting.
GitHub Actions: Your code lives on GitHub, team prefers GitHub's ecosystem, and you want multi-cloud flexibility.
GitLab CI: Your code lives on GitLab, you want self-hosted runners, or you need built-in security scanning (SAST/DAST).

Security in the Pipeline

Shift-Left Security

Stage	Tool	What It Catches
Pre-commit	git-secrets, talisman	Hardcoded credentials before they enter repo
Build	SAST (CodeGuru, Snyk, Semgrep)	Code vulnerabilities (SQL injection, XSS)
Build	SCA (Dependabot, Snyk)	Vulnerable dependencies
Container Build	ECR image scanning, Trivy	Container CVEs
Pre-Deploy	IAM policy validation (IAM Access Analyzer)	Over-privileged roles
Post-Deploy	DAST (OWASP ZAP)	Runtime vulnerabilities

Pipeline Security Best Practices

No long-lived credentials — use OIDC (GitHub) or IAM roles (CodeBuild)
Least privilege — pipeline role can only deploy to specific services/accounts
Artifact signing — sign container images (cosign / Notation) to ensure integrity
Immutable artifacts — tag images with git SHA, never overwrite :latest in production
Approval gates — require human approval before production deployment
Audit trail — CloudTrail logs all deployment actions

Deployment Observability

A deployment isn't done when the pipeline turns green. Monitor after deploy:

Post-Deploy Validation

Deploy v2 → Wait 5 min → Check:
  ├── Error rate increased? → Rollback
  ├── Latency p99 > threshold? → Rollback
  ├── Health check failures? → Rollback
  └── All green → Deployment successful

CloudWatch Alarms as Deployment Gates

Configure CodeDeploy / ECS circuit breaker to monitor CloudWatch alarms:

If alarm triggers during canary/linear deployment → automatic rollback
No human intervention needed for obvious failures

Common Pipeline Anti-Patterns

Anti-Pattern	Problem	Fix
No staging environment	Bugs found in production	Always deploy to staging first
Tests only in CI, not in deployment	Broken integration passes build	Run integration tests post-deploy
Manual deployment to production	Error-prone, unauditable	Automate everything, gate with approval
Same pipeline for all environments	No quality gates between stages	Multi-stage with promotion (dev → staging → prod)
`:latest` tag in production	Can't tell which version is running	Use git SHA or semantic version tags
Secrets in pipeline code	Credential exposure	Use OIDC, Secrets Manager, or Parameter Store
No rollback plan	Stuck with broken deployment	Blue/green or canary with auto-rollback

Summary

CI/CD on AWS comes down to three decisions:

Deployment strategy — Rolling (simple), Blue/Green (zero-downtime), Canary (lowest risk), or Feature Flags (decouple deploy from release)
Pipeline tool — CodePipeline (AWS-native, deep integration), GitHub Actions (flexible, multi-cloud), or GitLab CI (self-hosted, built-in security)
Safety mechanisms — Approval gates, CloudWatch alarm-based rollback, immutable artifacts, shift-left security scanning

The 2026 default for containers: ECS native blue/green with canary traffic shifting + configurable circuit breaker. No CodeDeploy needed for standard ECS workloads.

The 2026 default for pipeline: GitHub Actions with OIDC auth for most teams. CodePipeline when you need native cross-account patterns or CodeDeploy's advanced strategies.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS infrastructure automation and DevOps practices. Connect on LinkedIn.

AWS Database Selection Guide: RDS vs Aurora vs DynamoDB vs ElastiCache — Choosing the Right Database for Every Workload

AlpeshKumbhare — Mon, 17 Aug 2026 05:54:18 +0000

AWS offers 15+ managed database services. Each is purpose-built for specific data models and access patterns. Choosing wrong means either fighting the database's design (performance problems) or over-engineering a simple workload (cost problems).

The key insight: start with your access patterns, not the database. How will data be read and written? What queries are critical? What's the read/write ratio? The answers point directly to the right service.

This guide covers every AWS database service, organized by data model, with a decision framework that gets you to the right choice fast.

The AWS Database Landscape

┌─────────────────────────────────────────────────────────────────────┐
│                       AWS DATABASE SERVICES                           │
├─────────────────────────────────────────────────────────────────────┤
│  RELATIONAL (SQL)                                                    │
│  RDS (MySQL, PostgreSQL, Oracle, SQL Server, MariaDB) | Aurora       │
├─────────────────────────────────────────────────────────────────────┤
│  KEY-VALUE / DOCUMENT (NoSQL)                                        │
│  DynamoDB                                                            │
├─────────────────────────────────────────────────────────────────────┤
│  IN-MEMORY (Caching / Real-Time)                                     │
│  ElastiCache (Redis/Valkey, Memcached) | MemoryDB for Redis          │
├─────────────────────────────────────────────────────────────────────┤
│  GRAPH                                                               │
│  Neptune                                                             │
├─────────────────────────────────────────────────────────────────────┤
│  TIME SERIES                                                         │
│  Timestream                                                          │
├─────────────────────────────────────────────────────────────────────┤
│  WIDE COLUMN                                                         │
│  Keyspaces (Cassandra-compatible)                                    │
├─────────────────────────────────────────────────────────────────────┤
│  SEARCH                                                              │
│  OpenSearch Service                                                  │
├─────────────────────────────────────────────────────────────────────┤
│  LEDGER                                                              │
│  QLDB (Quantum Ledger Database)                                      │
├─────────────────────────────────────────────────────────────────────┤
│  DATA WAREHOUSE                                                      │
│  Redshift                                                            │
├─────────────────────────────────────────────────────────────────────┤
│  VECTOR (AI/ML)                                                      │
│  Aurora (pgvector) | OpenSearch | MemoryDB | Neptune Analytics        │
└─────────────────────────────────────────────────────────────────────┘

The Decision Flowchart

START
│
├── Need SQL, joins, transactions, complex queries?
│     ├── High performance, auto-scaling, global? → Aurora
│     └── Standard workload, specific engine (Oracle/SQL Server)? → RDS
│
├── Need key-value or simple document access at any scale?
│     └── DynamoDB
│
├── Need microsecond latency caching?
│     ├── Cache-aside pattern (volatile)? → ElastiCache
│     └── Need durability (primary datastore)? → MemoryDB
│
├── Need relationship traversal (social graph, fraud, recommendations)?
│     └── Neptune
│
├── Need time-series data (IoT, metrics, logs)?
│     └── Timestream
│
├── Need full-text search, log analytics?
│     └── OpenSearch
│
├── Need analytics on petabytes (BI, reporting)?
│     └── Redshift
│
├── Need vector similarity search (AI/RAG)?
│     └── Aurora pgvector / OpenSearch / MemoryDB
│
└── Need immutable, cryptographically verifiable ledger?
      └── QLDB

Relational Databases: RDS vs Aurora

Amazon RDS

What: Managed relational database for MySQL, PostgreSQL, MariaDB, Oracle, SQL Server.

AWS manages: Patching, backups, Multi-AZ failover, monitoring.
You manage: Instance sizing, parameter tuning, schema design, query optimization.

Key features:

Multi-AZ: synchronous standby for HA (automatic failover ~60 seconds)
Read Replicas: up to 15 (async replication) for read scaling
Automated backups: point-in-time recovery (35-day window)
Storage auto-scaling: up to 64 TiB

Amazon Aurora

What: AWS-rebuilt MySQL/PostgreSQL with cloud-native storage architecture. Compatible at the wire-protocol level (drop-in replacement).

Why Aurora over RDS:

Feature	RDS	Aurora
Performance	1x (standard engine)	3-5x MySQL, 3x PostgreSQL
Storage	EBS (single AZ)	Distributed across 3 AZs (6 copies)
Storage limit	64 TiB	128 TiB (auto-grows)
Failover	~60 seconds	~30 seconds (with Aurora Replicas)
Read replicas	Up to 15 (async)	Up to 15 (same storage, <10ms lag)
Backtrack	❌	✅ (rewind to point in time without restore)
Global Database	❌	✅ (<1 second cross-region replication)
Serverless	❌	✅ (Aurora Serverless v2 — scales to zero)
Multi-master	❌	✅ (Aurora DSQL — distributed SQL, new)

When to Choose RDS vs Aurora

Choose RDS when...	Choose Aurora when...
Need Oracle or SQL Server	MySQL or PostgreSQL workload
Budget-constrained (RDS is cheaper for small instances)	Need high availability (<30s failover)
Simple, low-traffic application	Read-heavy workload (leverage replicas on shared storage)
Specific engine version required	Need global database (cross-region DR)
	Need serverless (variable/unpredictable traffic)
	Need to scale storage beyond 64 TiB

Aurora Serverless v2

Scales compute automatically (0.5 to 256 ACUs) based on demand. You pay per ACU-hour consumed.

Best for: Dev/test environments, variable workloads, new applications with unknown traffic patterns.

Not for: Steady high-traffic production (provisioned instances are cheaper when utilization is consistently high).

DynamoDB: NoSQL at Any Scale

What: Fully managed key-value and document database. Single-digit millisecond performance at any scale.

When to Use DynamoDB

Key-value or simple document access patterns
Extreme scale (millions of requests/second)
Predictable, consistent latency requirements
Serverless architecture (scales to zero with on-demand mode)
Global applications (Global Tables for multi-region active-active)

When NOT to Use DynamoDB

Complex queries with joins, aggregations, ad-hoc SQL
Data model that requires normalization and relational integrity
Analytics/reporting (use Redshift or Athena instead)
Small dataset with complex query needs (Aurora is simpler)

DynamoDB Key Design Decisions

Decision	Options
Capacity mode	On-Demand (unpredictable traffic, pay per request) vs Provisioned (predictable, cheaper at scale)
Primary key	Partition key only (unique lookup) vs Partition + Sort key (range queries within partition)
Global Tables	Multi-region active-active replication (near-zero RPO)
DAX	In-memory cache for DynamoDB (microsecond reads for hot data)
Streams	Capture changes for event-driven processing (CDC)

DynamoDB Pricing Reality

On-Demand: $1.25 per million write requests, $0.25 per million reads
Provisioned: ~$0.00065 per WCU/hour, ~$0.00013 per RCU/hour

Storage: $0.25/GB/month

Example: 10M reads + 1M writes per day
On-Demand: ~$10/day = $300/month
Provisioned: ~$120/month (60% cheaper at consistent load)

ElastiCache vs MemoryDB: Caching and Real-Time

ElastiCache (Redis/Valkey or Memcached)

Use for: Caching layer in front of databases. Session storage. Leaderboards. Real-time analytics.

Feature	Redis/Valkey	Memcached
Data structures	Rich (strings, hashes, lists, sets, sorted sets)	Simple key-value only
Persistence	Optional (snapshot + AOF)	None (volatile)
Replication	Yes (read replicas, Multi-AZ)	No replication
Pub/Sub	Yes	No
Cluster mode	Yes (horizontal sharding)	Yes (simple sharding)
Use case	Primary cache + data structures	Simple caching only

MemoryDB for Redis

What: Redis-compatible, durable, in-memory database. Unlike ElastiCache (which is a cache), MemoryDB is a primary database that happens to be in-memory.

Key difference: MemoryDB writes to a distributed transaction log before acknowledging — data survives node failures. ElastiCache can lose data on failover.

Use MemoryDB when: You need microsecond reads AND durability (session store as source of truth, real-time user profiles, gaming state).

Purpose-Built Databases

Amazon Neptune (Graph)

Use for: Relationship-heavy data where traversals are the primary query pattern.

Social networks (friend-of-friend queries)
Fraud detection (link analysis)
Recommendation engines (collaborative filtering)
Knowledge graphs (entity relationships)
Network topology (impact analysis)

NOT for: Simple lookups, transactional data, analytics. If you're not doing multi-hop traversals, you don't need a graph database.

Amazon Timestream (Time Series)

Use for: Time-stamped data that's primarily queried by time range.

IoT sensor data
Application/infrastructure metrics
Financial market data (ticks)
Fleet/device telemetry

Why over DynamoDB: Built-in time-based retention tiers (hot → cold → archive), time-series functions (interpolation, smoothing, aggregation), and automatic data lifecycle management.

Amazon Keyspaces (Cassandra)

Use for: Teams with existing Cassandra workloads wanting managed service. Wide-column data model for write-heavy workloads with predictable access patterns.

Amazon OpenSearch (Search + Analytics)

Use for: Full-text search, log analytics, application search, observability data.

Powers CloudWatch Log Insights behind the scenes
Kibana/OpenSearch Dashboards for visualization
Vector search for AI/RAG applications

Amazon QLDB (Ledger)

Use for: Immutable, cryptographically verifiable transaction history.

Financial transactions requiring audit trail
Supply chain provenance tracking
Regulatory compliance (tamper-proof records)

Amazon Redshift (Data Warehouse)

Use for: Analytical queries on large datasets (petabytes). BI reporting, data warehousing, complex aggregations across millions of rows.

NOT for: OLTP (transactional workloads) — use Aurora or DynamoDB instead.

Vector Databases for AI/RAG

With generative AI, vector similarity search is a new access pattern. AWS options:

Service	Vector Capability	Best For
Aurora PostgreSQL (pgvector)	Extension on existing Aurora	Teams already on Aurora, moderate scale
OpenSearch	k-NN plugin	Large-scale similarity search + hybrid text/vector
MemoryDB	Vector search support	Ultra-low latency vector retrieval
Neptune Analytics	Graph + vector	Knowledge graph with semantic search
Bedrock Knowledge Base	Managed (any backend)	Easiest path — managed vector storage

Multi-Database Architecture Patterns

Pattern 1: CQRS (Command Query Responsibility Segregation)

Writes → Aurora (source of truth, transactions)
Reads → ElastiCache (cached hot data) + DynamoDB (materialized views)

Pattern 2: Event Sourcing + Materialized Views

Events → DynamoDB (event store) → Streams → Lambda → 
  ├── OpenSearch (search index)
  ├── ElastiCache (real-time aggregations)
  └── Redshift (analytics)

Pattern 3: Polyglot Persistence

User Profiles → DynamoDB (key-value, high scale)
Product Catalog → Aurora (relational, complex queries)
Recommendations → Neptune (graph traversal)
Session State → ElastiCache (microsecond, volatile)
Search → OpenSearch (full-text)
Analytics → Redshift (OLAP, BI)

Common Mistakes

Mistake	Problem	Fix
DynamoDB for ad-hoc SQL queries	Expensive scans, poor performance	Use Aurora or Athena
Aurora for simple key-value access	Over-engineered, slower than DynamoDB	Use DynamoDB
ElastiCache as primary datastore	Data loss on failover	Use MemoryDB if durability needed
Single database for everything	One size fits none	Polyglot persistence (right DB per access pattern)
Redshift for OLTP	Terrible latency for single-row reads	Use Aurora or DynamoDB
Ignoring connection pooling (Aurora)	Connection exhaustion under load	Use RDS Proxy
DynamoDB without understanding key design	Hot partitions, throttling	Design partition key for even distribution

Cost Comparison (Typical Web App)

Service	Starting Cost (dev)	Production Estimate
RDS db.t4g.medium	~$50/month	~$200-500/month (Multi-AZ)
Aurora Serverless v2 (min 0.5 ACU)	~$45/month	Scales with usage
Aurora Provisioned (db.r6g.large)	~$180/month	~$400-800/month (Multi-AZ + replicas)
DynamoDB (On-Demand, light)	~$5-25/month	Scales per request
ElastiCache (cache.t4g.small)	~$25/month	~$100-300/month (Multi-AZ)
OpenSearch (t3.small.search)	~$35/month	~$200-500/month (Multi-AZ)

Summary

AWS database selection is driven by access patterns, not features:

Access Pattern	Database
SQL, joins, transactions, complex queries	Aurora (or RDS for Oracle/SQL Server)
Key-value lookups at massive scale	DynamoDB
Microsecond caching (volatile)	ElastiCache
Microsecond reads (durable)	MemoryDB
Relationship traversal (graph)	Neptune
Time-series data	Timestream
Full-text search	OpenSearch
Analytical queries (BI)	Redshift
Vector similarity (AI/RAG)	Aurora pgvector or OpenSearch
Immutable audit ledger	QLDB

The guiding principle: One database rarely fits all access patterns. Use purpose-built databases for each pattern (polyglot persistence), connected by event-driven synchronization.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS data architecture and cloud infrastructure automation. Connect on LinkedIn.

Event-Driven Architecture on AWS: SQS vs SNS vs EventBridge vs Kinesis — Patterns, Anti-Patterns, and When to Use What

AlpeshKumbhare — Fri, 14 Aug 2026 05:38:42 +0000

Event-driven architecture (EDA) is how modern distributed systems communicate at scale. Instead of services calling each other directly (tight coupling), services emit events ("this happened") and interested consumers react independently.

On AWS, four messaging services form the EDA backbone — but they solve different problems. Choosing wrong means either over-engineering a simple notification into a Kinesis stream, or under-engineering a high-throughput data pipeline onto SQS.

This guide maps each service to its sweet spot, covers the integration patterns that work in production, and highlights the anti-patterns that waste money and create operational pain.

The Four Messaging Services

┌──────────────────────────────────────────────────────────────────────┐
│                    AWS MESSAGING LANDSCAPE                             │
├──────────────┬──────────────┬───────────────────┬────────────────────┤
│     SQS      │     SNS      │   EventBridge     │     Kinesis        │
│              │              │                   │                    │
│  Queue       │  Pub/Sub     │  Event Bus        │  Stream            │
│  (1:1)       │  (1:many)    │  (content-route)  │  (ordered, replay) │
│              │              │                   │                    │
│  Decouple    │  Fan-out     │  Route + Filter   │  Real-time data    │
│  + buffer    │  broadcast   │  + transform      │  high throughput   │
└──────────────┴──────────────┴───────────────────┴────────────────────┘

Quick Decision Matrix

If you need...	Use...
Decouple producer/consumer, buffer load	SQS
Send one event to many subscribers	SNS
Route events based on content/attributes	EventBridge
Receive events from SaaS (Stripe, Auth0, Shopify)	EventBridge
Process ordered, replayable data stream	Kinesis Data Streams
High-throughput ingestion (100K+ events/sec)	Kinesis
Fan-out + per-consumer buffering	SNS → SQS (combined)
Transform/enrich events between services	EventBridge Pipes
Schedule future events (cron)	EventBridge Scheduler

Amazon SQS: The Queue

What it is: Point-to-point message queue. One producer puts messages, one consumer processes them. Messages are buffered until consumed.

Two Flavors

Feature	Standard Queue	FIFO Queue
Ordering	Best-effort (may reorder)	Strict FIFO guaranteed
Deduplication	At-least-once (may duplicate)	Exactly-once processing
Throughput	Unlimited	3,000 msg/sec (with batching: 30,000)
Use case	High throughput, order doesn't matter	Financial transactions, command sequences

When to Use SQS

Load leveling — smooth bursty traffic (e.g., API receives 10K requests/sec, worker processes at 1K/sec)
Decoupling — producer doesn't need to know about consumer (or if it's running)
Retry/DLQ — failed messages automatically route to Dead Letter Queue for investigation
Batch processing — Lambda polls SQS, processes in batches of up to 10

SQS Architecture Pattern: Work Queue

┌──────────┐     ┌──────────┐     ┌──────────────┐
│  API GW  │────→│   SQS    │────→│ Lambda / ECS │
│  (burst) │     │  (buffer)│     │  (steady)    │
└──────────┘     └──────────┘     └──────────────┘
                       │
                       ▼ (after 3 failures)
                 ┌──────────┐
                 │   DLQ    │
                 └──────────┘

Amazon SNS: Pub/Sub Fan-Out

What it is: Publish-subscribe messaging. One publisher sends to a topic, multiple subscribers receive copies.

Subscriber Types

SQS queues (most common — adds buffering per consumer)
Lambda functions (direct invocation)
HTTP/HTTPS endpoints (webhooks)
Email / SMS (notifications)
Kinesis Data Firehose (streaming delivery)
Mobile push (iOS/Android)

When to Use SNS

Fan-out — one event needs to trigger multiple independent actions
Notifications — email alerts, SMS, mobile push
Decoupled fan-out — SNS → multiple SQS queues (each consumer has its own queue)

SNS + SQS: The Fan-Out Pattern

                         ┌──── SQS (Email Service) ──→ Lambda: send email
                         │
Producer ──→ SNS Topic ──┼──── SQS (Analytics) ──→ Lambda: track metrics
                         │
                         └──── SQS (Inventory) ──→ Lambda: update stock

Each consumer gets its own SQS queue with independent retry and DLQ. One consumer's failure doesn't affect others.

Amazon EventBridge: The Event Router

What it is: Serverless event bus with content-based routing. Events flow in, rules match patterns, and events route to targets — all without code.

Why EventBridge Over SNS

Feature	SNS	EventBridge
Filtering	Basic attribute filtering	Rich content-based rules (any JSON field)
Schema	None	Schema registry + discovery
SaaS integration	No	30+ SaaS partners (Stripe, Auth0, Zendesk)
Archive & replay	No	Yes (replay events from any point in time)
Cross-account	Complex	Native (event bus sharing)
Transforms	No	Input transformer (reshape events before delivery)
Pipes	No	EventBridge Pipes (filter → enrich → transform → deliver)
Pricing	Per publish + delivery	Per event ingested ($1/million)

When to Use EventBridge

Content-based routing — route events based on any field in the event body
SaaS events — receive events from third-party services without polling
Cross-account event sharing — centralized event bus for multi-account architectures
Event replay — archive events and replay when debugging or reprocessing
Schema enforcement — discover and validate event schemas automatically

EventBridge Rule Example

{
  "source": ["com.myapp.orders"],
  "detail-type": ["OrderCreated"],
  "detail": {
    "amount": [{"numeric": [">", 1000]}],
    "region": ["eu-west-1", "eu-central-1"]
  }
}

This rule matches only: orders > $1000 from EU regions. Everything else is ignored. No code needed.

EventBridge Architecture Pattern: Event Mesh

┌─────────────┐        ┌─────────────────┐        ┌─────────────┐
│ Order Service│──emit──→│  EventBridge    │──rule──→│ Payment Svc │
└─────────────┘        │  (Central Bus)  │        └─────────────┘
                        │                 │
┌─────────────┐        │  Rules:         │        ┌─────────────┐
│ Auth0 (SaaS)│──emit──→│  • OrderCreated │──rule──→│Warehouse Svc│
└─────────────┘        │  • UserSignedUp │        └─────────────┘
                        │  • PaymentFailed│
┌─────────────┐        │                 │        ┌─────────────┐
│ Stripe(SaaS)│──emit──→│                 │──rule──→│ Notification│
└─────────────┘        └─────────────────┘        └─────────────┘

Amazon Kinesis: The Data Stream

What it is: Real-time data streaming for high-throughput, ordered, replayable event processing.

Kinesis Family

Service	Purpose
Kinesis Data Streams	Custom real-time stream processing (you control consumers)
Kinesis Data Firehose	Managed delivery to S3, Redshift, OpenSearch (zero code)
Kinesis Data Analytics	SQL/Flink on streaming data (real-time analytics)

When to Use Kinesis Over SQS/EventBridge

Requirement	SQS/EventBridge	Kinesis
Ordering guarantee	FIFO SQS (limited)	Per-shard ordering (scalable)
Event replay	EventBridge archive	Native (24h-365d retention)
Multiple consumers on same stream	❌ (SNS fan-out)	✅ (multiple consumers, each at own position)
Throughput	Millions/sec (SQS)	1MB/sec per shard (scale shards)
Real-time analytics	Not designed for	Built for this
Use case	Application events, notifications	IoT telemetry, clickstream, logs, financial ticks

Kinesis Architecture Pattern: Real-Time Analytics

┌────────────┐     ┌─────────────┐     ┌───────────────────┐
│ IoT Devices│────→│  Kinesis     │────→│ Lambda (real-time) │
│ Clickstream│     │  Data Stream │     │ Anomaly detection  │
│ App Logs   │     │  (ordered)   │     └───────────────────┘
└────────────┘     └──────┬──────┘
                          │
                          ├────→ Firehose → S3 (data lake)
                          └────→ Flink (windowed aggregations)

EventBridge Pipes: Connect + Transform

EventBridge Pipes (launched 2023) connects sources to targets with optional filtering, enrichment, and transformation — without writing Lambda glue code.

Source → Filter → Enrich → Transform → Target

Example:
SQS Queue → filter (only "critical") → Lambda (add metadata) → reshape JSON → EventBridge Bus

Supported Sources & Targets

Sources: SQS, Kinesis, DynamoDB Streams, Kafka (MSK), Self-Managed Kafka

Targets: Lambda, Step Functions, ECS Task, EventBridge Bus, API Gateway, SQS, SNS, Kinesis, and more

When to Use Pipes vs Lambda

Scenario	Pipes	Lambda
Filter + route (no business logic)	✅ No code	Overkill
Simple field transformation	✅ Input transformer	Overkill
Complex business logic	❌	✅ Needed
Enrich from external API	✅ (enrichment step)	Also works

EventBridge Scheduler

For time-based events, EventBridge Scheduler replaces CloudWatch Events (cron):

One-time schedules — "send reminder email in 48 hours"
Recurring schedules — "run cleanup every day at 2 AM"
Rate-based — "trigger every 5 minutes"
Timezone-aware — handles DST correctly (CloudWatch Events doesn't)
At-scale — millions of individual schedules (one per user/order/entity)

// Schedule a one-time future event
{
  "ScheduleExpression": "at(2026-08-20T14:00:00)",
  "Target": {
    "Arn": "arn:aws:lambda:...:send-reminder",
    "Input": "{\"orderId\": \"order-123\", \"action\": \"follow-up\"}"
  }
}

Integration Patterns

Pattern 1: Command Queue (SQS)

Use for: async task processing, work distribution

API → SQS → Worker (Lambda/ECS)

Pattern 2: Fan-Out (SNS → SQS)

Use for: one event, multiple independent reactions

Service → SNS → SQS(A) → Consumer A
                → SQS(B) → Consumer B
                → SQS(C) → Consumer C

Pattern 3: Event Router (EventBridge)

Use for: content-based routing, SaaS integration, cross-account

Services → EventBridge Bus → Rules → Targets (Lambda, SQS, Step Functions)

Pattern 4: Streaming Pipeline (Kinesis)

Use for: real-time ordered data, multiple consumers, replay

Producers → Kinesis → Consumer A (real-time alerts)
                    → Consumer B (S3 archive via Firehose)
                    → Consumer C (analytics via Flink)

Pattern 5: Choreography (EventBridge, cross-service)

Use for: loosely coupled microservice workflows

Order Created → (event) → Payment Service reacts
Payment Succeeded → (event) → Shipping Service reacts
Shipping Completed → (event) → Notification Service reacts

No orchestrator. Each service reacts independently to relevant events.

Pattern 6: Orchestration + Events (Step Functions + EventBridge)

Use for: complex workflows with visibility + event-driven triggers

EventBridge detects event → triggers Step Function →
Step Function orchestrates multi-step workflow →
emits completion event back to EventBridge

Anti-Patterns to Avoid

Anti-Pattern	Problem	Fix
Using Kinesis for simple notifications	Expensive, complex for low throughput	SQS or SNS
SQS for fan-out (multiple consumers)	Only one consumer per message	SNS → SQS pattern
EventBridge for 100K+ events/sec	Soft limits, cost adds up	Kinesis for high throughput
Synchronous chains disguised as events	Hidden coupling, hard to debug	True async with DLQs
No Dead Letter Queue	Lost messages on failure	Always configure DLQ
Giant event payloads (>256KB)	Exceeds size limits	Store payload in S3, pass reference
No event schema/contract	Breaking consumers on format change	EventBridge Schema Registry
Ignoring idempotency	Duplicate processing causes data corruption	Design consumers to be idempotent

Cost Comparison

Service	Pricing Model	Cost at 10M events/month
SQS Standard	$0.40/million requests	~$4
SQS FIFO	$0.50/million requests	~$5
SNS	$0.50/million publishes + delivery costs	~$5-15
EventBridge	$1.00/million events	~$10
Kinesis (1 shard)	$0.015/hr + $0.014/million PUT	~$15
Kinesis (10 shards)	$0.15/hr + $0.014/million PUT	~$110

Takeaway: SQS is cheapest for simple queuing. EventBridge costs 2.5x SQS but provides routing, filtering, schema, archive, and replay. Kinesis is the most expensive but provides ordering, replay, and real-time stream processing.

Summary

Event-driven architecture on AWS comes down to four services with distinct strengths:

SQS — queue + buffer. Decouple producer and consumer. Simple, cheap, reliable.
SNS — fan-out. One event to many subscribers. Combine with SQS for durability.
EventBridge — intelligent routing. Content-based filtering, SaaS integration, archive/replay, cross-account. The "default choice" for new EDA designs.
Kinesis — ordered stream. High throughput, multiple consumers, real-time processing. Use for data pipelines, not application events.

The default starting point in 2026: EventBridge for routing + SQS for buffering. Add Kinesis only when you need ordering, replay at volume, or real-time analytics. Use SNS when you need mobile push or email notifications.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS infrastructure automation and event-driven cloud architecture. Connect on LinkedIn.

AWS Observability in 2026: The Complete Stack — CloudWatch, X-Ray, OpenTelemetry, and Application Signals

AlpeshKumbhare — Thu, 13 Aug 2026 09:11:41 +0000

You can't fix what you can't see. As systems grow more distributed — microservices, serverless functions, event-driven flows — the gap between "deployed" and "observable" becomes the difference between resolving incidents in minutes versus hours.

AWS's observability stack has evolved significantly. X-Ray SDK entered maintenance mode in February 2026, replaced by OpenTelemetry. Application Signals introduced SLO-based monitoring. CloudWatch absorbed capabilities that previously required third-party tools.

This post covers the complete AWS observability architecture for modern applications — what to collect, where to send it, and how to build alerting that actually works.

The Three Pillars of Observability

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   METRICS   │    │    LOGS     │    │   TRACES    │
│             │    │             │    │             │
│ What's      │    │ What        │    │ Where in    │
│ happening?  │    │ happened?   │    │ the path?   │
│             │    │             │    │             │
│ CPU, memory │    │ Error msgs  │    │ Request     │
│ request rate│    │ debug info  │    │ flow across │
│ error count │    │ audit trail │    │ services    │
│ latency p99 │    │             │    │             │
└─────────────┘    └─────────────┘    └─────────────┘
      │                   │                   │
      └───────────────────┼───────────────────┘
                          │
                    ┌─────────────┐
                    │ CORRELATION │
                    │ Trace ID →  │
                    │ connects    │
                    │ all three   │
                    └─────────────┘

Metrics tell you WHAT is wrong (error rate spiked).
Logs tell you WHY it's wrong (NullPointerException at line 42).
Traces tell you WHERE in the request path it went wrong (payment-service → database timeout).

All three need a common correlation key (trace ID) to be useful together.

The AWS Observability Stack

┌─────────────────────────────────────────────────────────────────┐
│  COLLECTION (Instrumentation)                                    │
│  ADOT (OpenTelemetry) | CloudWatch Agent | SDKs                  │
├─────────────────────────────────────────────────────────────────┤
│  PROCESSING & STORAGE                                            │
│  CloudWatch (Metrics + Logs) | X-Ray (Traces) | S3 (Archive)    │
├─────────────────────────────────────────────────────────────────┤
│  ANALYSIS                                                        │
│  Log Insights | Metrics Insights | Trace Analytics | ServiceLens │
├─────────────────────────────────────────────────────────────────┤
│  SLO & ALERTING                                                  │
│  Application Signals | CloudWatch Alarms | Composite Alarms      │
├─────────────────────────────────────────────────────────────────┤
│  VISUALIZATION                                                   │
│  CloudWatch Dashboards | ServiceLens Map | Managed Grafana        │
└─────────────────────────────────────────────────────────────────┘

Metrics: CloudWatch Metrics + EMF

Default Metrics (Free)

AWS services automatically emit metrics to CloudWatch:

EC2: CPU, network, disk, status checks
ECS/Fargate: CPU, memory utilization per task
Lambda: invocations, duration, errors, throttles, cold starts
ALB: request count, latency, HTTP 4xx/5xx, healthy host count
RDS: connections, IOPS, replication lag, free storage

Custom Metrics

For application-specific metrics, use Embedded Metric Format (EMF):

{
  "_aws": {
    "Timestamp": 1692286800000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp/Orders",
      "Dimensions": [["Service", "Environment"]],
      "Metrics": [
        {"Name": "OrdersProcessed", "Unit": "Count"},
        {"Name": "ProcessingTime", "Unit": "Milliseconds"}
      ]
    }]
  },
  "Service": "order-service",
  "Environment": "production",
  "OrdersProcessed": 47,
  "ProcessingTime": 230
}

Why EMF over PutMetricData: EMF lets you log structured JSON that CloudWatch automatically extracts as metrics AND preserves as log entries — one write, two outputs. No PutMetricData API calls (cheaper, lower latency).

Container Insights

For ECS and EKS, Container Insights provides:

Per-container CPU and memory
Per-pod/task network I/O
Cluster-level resource utilization
Kubernetes-aware dimensions (namespace, deployment, pod)

Enable with: ADOT collector as DaemonSet (EKS) or sidecar (ECS).

Logs: CloudWatch Logs + Log Insights

Structured Logging (Essential)

Unstructured logs are unsearchable at scale. Always log structured JSON:

{
  "timestamp": "2026-08-13T10:30:00Z",
  "level": "ERROR",
  "service": "payment-service",
  "traceId": "1-abc123-def456",
  "requestId": "req-789",
  "message": "Payment failed",
  "error": "TimeoutException",
  "customerId": "cust-123",
  "amount": 99.99,
  "duration_ms": 5002
}

Log Insights Queries

CloudWatch Log Insights provides SQL-like querying across log groups:

# Find slowest requests in the last hour
fields @timestamp, service, duration_ms, requestId
| filter duration_ms > 1000
| sort duration_ms desc
| limit 20

# Error rate per service
filter level = "ERROR"
| stats count(*) as errors by service
| sort errors desc

# Trace specific request across services
filter traceId = "1-abc123-def456"
| sort @timestamp asc

Log Architecture Patterns

Pattern	When
Direct to CloudWatch	Lambda (automatic), ECS (awslogs driver), EKS (Fluent Bit)
CloudWatch → S3	Long-term retention, compliance archives (subscription filter)
CloudWatch → OpenSearch	Need full-text search, complex aggregations, Kibana dashboards
ADOT → CloudWatch	OpenTelemetry-based collection with CloudWatch backend

Retention and Cost

Set retention policies per log group (don't default to "never expire")
Production: 30-90 days in CloudWatch, archive to S3 Glacier after
Dev/staging: 7-14 days (no archival)
Infrequent Access class: 50% cheaper for logs you rarely query

Traces: X-Ray + OpenTelemetry (ADOT)

The Migration: X-Ray SDK → OpenTelemetry

Important: X-Ray SDK entered maintenance mode in February 2026. AWS now recommends OpenTelemetry for all new instrumentation.

Old Approach	New Approach
X-Ray SDK (language-specific)	AWS Distro for OpenTelemetry (ADOT)
Proprietary trace format	OpenTelemetry (OTel) standard
AWS-only export	Export to X-Ray, Jaeger, Zipkin, Grafana Tempo, etc.
Auto-instrumentation (limited)	Auto-instrumentation (comprehensive)

ADOT Architecture

┌────────────┐     ┌──────────────┐     ┌─────────────┐
│Application │────→│ ADOT Collector│────→│  X-Ray      │
│(OTel SDK)  │     │ (sidecar or  │     │  CloudWatch │
│            │     │  DaemonSet)  │     │  Prometheus │
└────────────┘     └──────────────┘     └─────────────┘

ADOT Collector receives traces/metrics via OTLP protocol and exports to one or more backends. You can send to X-Ray AND Grafana Tempo simultaneously.

Auto-Instrumentation (Zero Code Changes)

For Java, Python, Node.js, and .NET — ADOT auto-instrumentation captures:

HTTP requests (incoming and outgoing)
Database queries (SQL, DynamoDB, Redis)
AWS SDK calls (S3, SQS, SNS, Lambda invocations)
gRPC calls

EKS: Deploy ADOT auto-instrumentation as a Kubernetes operator — injects instrumentation into pods automatically.

Lambda: Enable Lambda X-Ray Active Tracing — one toggle, zero code.

Trace Anatomy

Trace: 1-abc123-def456 (entire request lifecycle)
│
├── Span: API Gateway (12ms)
│     └── Span: Lambda: order-handler (450ms)
│           ├── Span: DynamoDB: GetItem (23ms)
│           ├── Span: HTTP: payment-service (380ms)
│           │     └── Span: RDS: INSERT (45ms)  ← SLOW?
│           └── Span: SQS: SendMessage (15ms)

Each span shows: service name, duration, status, metadata. Find the bottleneck instantly.

Application Signals: SLO-Based Monitoring

Application Signals (GA 2025) is CloudWatch's answer to "monitor what matters to users" — automatically tracks SLIs (Service Level Indicators) and lets you define SLOs (Service Level Objectives).

What It Auto-Discovers

Without any configuration, Application Signals detects:

Services and their dependencies (service map)
Call volume between services
Latency (p50, p90, p99)
Error rate and fault rate
Availability

Defining SLOs

SLO: "Payment Service Availability"
├── SLI: Success rate (HTTP 2xx / total requests)
├── Target: 99.9% over 30-day rolling window
├── Error budget: 0.1% (43 minutes/month of allowed errors)
└── Alert: When burn rate exceeds 10x normal → page on-call

Why SLOs > Threshold Alarms

Traditional Alarm	SLO-Based Alert
"CPU > 80%" → alert	"Error budget burning too fast" → alert
Noisy, often false positive	Only fires when users are impacted
Measures infrastructure health	Measures user experience
Doesn't account for context	Accounts for error budget remaining

Alerting That Works

Alarm Strategy

Layer	What to Alert On	Action
SLO breach	Error budget burn rate > threshold	Page on-call immediately
Service health	Error rate > 5% for 5 minutes	Page on-call
Saturation	CPU > 85%, memory > 90%, disk > 80%	Auto-scale + notify
Dependencies	Upstream latency > SLA	Notify (not page)
Business metrics	Orders/min drops > 50%	Alert business + engineering

Composite Alarms

Reduce noise by combining related alarms:

CompositeAlarm: "Payment Service Degraded"
├── AND: Error rate > 5%
├── AND: Latency p99 > 2000ms
└── AND: NOT in maintenance window

→ Only fires when BOTH error rate AND latency are degraded
→ Eliminates false positives from single-metric spikes

Alert Routing

CloudWatch Alarm → SNS Topic → Multiple targets:
├── PagerDuty/Opsgenie (critical: pages on-call)
├── Slack channel (warning: notification only)
├── Lambda (auto-remediation for known issues)
└── ITSM (ServiceNow incident creation)

Dashboards: What to Show

Per-Service Dashboard (Auto-Generated with Application Signals)

Request rate (rpm)
Error rate (%)
Latency (p50, p90, p99)
Dependency health (downstream services)
Recent deployments (correlate changes with metrics)

Platform Dashboard (SRE/Platform Team)

SLO status across all services (green/yellow/red)
Error budget remaining per service
Top 5 highest-latency services
Recent alerts and resolution time
Deployment frequency and failure rate

Cost Dashboard (FinOps)

Compute spend trend (CloudWatch billing metric)
Data transfer costs by service
Lambda invocations/cost correlation
Over-provisioned resources (Compute Optimizer findings)

Observability for Different Architectures

Serverless (Lambda + API Gateway + DynamoDB)

Pillar	Tool	Setup
Metrics	CloudWatch (auto) + EMF for custom	Lambda Powertools library
Logs	CloudWatch Logs (auto)	Structured JSON, set retention
Traces	X-Ray Active Tracing (one toggle)	Zero code
SLOs	Application Signals	Auto-discovers Lambda services

Containers (ECS/EKS + ALB + RDS)

Pillar	Tool	Setup
Metrics	Container Insights + EMF	ADOT collector as sidecar/DaemonSet
Logs	CloudWatch (awslogs/Fluent Bit)	Structured JSON, per-container log groups
Traces	ADOT (OpenTelemetry)	Auto-instrumentation operator (EKS)
SLOs	Application Signals	Service map auto-discovery

Hybrid (Containers + Lambda + Step Functions)

Pillar	Tool	Setup
Metrics	Container Insights + Lambda metrics	Unified CloudWatch namespace
Logs	CloudWatch Logs (all services)	Common trace ID in all log entries
Traces	ADOT + X-Ray Active Tracing	Trace propagation across Lambda→ECS→SQS
SLOs	Application Signals	End-to-end service map

Cost Optimization for Observability

Observability itself can be expensive. Control costs:

Lever	Savings
Log retention policies (don't keep forever)	50-80% on log storage
Infrequent Access log class	50% for rarely-queried logs
EMF instead of PutMetricData API	Avoid per-metric API charges
Sampling traces (e.g., 10% of requests)	90% trace storage savings
Metric filters instead of full log queries	Reduce Log Insights costs
Archive to S3 after 30 days	CloudWatch → S3 Glacier (95% cheaper)

Summary

AWS observability in 2026 centers on three shifts:

X-Ray SDK → OpenTelemetry (ADOT) — industry-standard instrumentation, multi-backend export, auto-instrumentation for zero code changes
Threshold alarms → SLO-based alerting — Application Signals measures what users experience, not just infrastructure health
Separate tools → unified correlation — trace IDs connect metrics, logs, and traces so you move from "something's wrong" to "here's the failing span" in seconds

The architecture: Instrument with ADOT (OTel), store in CloudWatch + X-Ray, analyze with Log Insights + Trace Analytics, alert on SLOs via Application Signals, visualize in CloudWatch Dashboards or Managed Grafana.

Start with auto-instrumentation and structured logs. Add custom metrics and SLOs as your services mature. Don't over-instrument day one — observability should grow with your system's complexity.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS observability, infrastructure automation, and cloud architecture. Connect on LinkedIn.

Terraform vs AWS CDK vs CloudFormation: The Definitive IaC Decision Guide for 2026

AlpeshKumbhare — Wed, 12 Aug 2026 07:35:58 +0000

Every AWS team eventually asks: "Should we use Terraform, CDK, or CloudFormation?" The answer isn't universal — it depends on your team's skills, organizational constraints, multi-cloud requirements, and how you want to manage infrastructure lifecycle.

This guide compares all three across the dimensions that actually matter in production, with a decision framework to help you choose.

The Three Models at a Glance

┌───────────────────────────────────────────────────────────────────┐
│                    INFRASTRUCTURE AS CODE                           │
├───────────────────┬───────────────────┬───────────────────────────┤
│   CloudFormation  │     AWS CDK       │       Terraform           │
│                   │                   │                           │
│   Declarative     │   Imperative      │   Declarative             │
│   JSON/YAML       │   TypeScript/     │   HCL                    │
│                   │   Python/Java/Go  │                           │
│   AWS-native      │   Synthesizes to  │   Multi-cloud             │
│                   │   CloudFormation   │   (4000+ providers)       │
│   No state file   │   No state file   │   State file required     │
│   (AWS manages)   │   (AWS manages)   │   (you manage)            │
└───────────────────┴───────────────────┴───────────────────────────┘

CloudFormation: The AWS Native

What it is: AWS's own IaC service. Declarative JSON/YAML templates that describe desired state. AWS handles provisioning, ordering, and rollback.

Strengths

Zero setup — no tools to install, no state to manage
Same-day AWS support — new services/features available immediately in CloudFormation
Drift detection — detects when resources deviate from template
Stack operations — create, update, delete as atomic operations with automatic rollback
StackSets — deploy across multiple accounts and regions from one template
Change sets — preview changes before applying
No state file — AWS tracks resource state internally (no S3 backend, no locking concerns)

Weaknesses

Verbose — simple resources require many lines of YAML/JSON
No loops or conditionals (limited) — Conditions and Fn::ForEach are awkward
No abstraction — can't create reusable "classes" of infrastructure
AWS-only — cannot manage non-AWS resources
Slow updates — large stacks take 30+ minutes to update
Error messages — often cryptic, debugging is painful

Best For

Teams deeply committed to AWS with no multi-cloud plans
Organizations using AWS Service Catalog (backed by CloudFormation)
Landing Zone Accelerator (LZA) deployments
Simple, single-account deployments with < 50 resources

AWS CDK: Infrastructure in Real Code

What it is: An open-source framework that lets you define infrastructure using programming languages (TypeScript, Python, Java, Go, C#). CDK synthesizes to CloudFormation — it's an abstraction layer on top.

Strengths

Real programming languages — loops, conditionals, functions, classes, inheritance
Constructs — reusable, composable building blocks (L1, L2, L3)
Type safety — IDE autocomplete, compile-time checks, refactoring support
Abstraction — define a "SecureWebApp" construct once, reuse everywhere
Testing — unit test infrastructure with standard testing frameworks (Jest, pytest)
Same deployment model as CloudFormation — benefits from AWS-managed state, rollback, drift detection
Construct Hub — community library of pre-built patterns (aws-solutions-constructs)

Weaknesses

Still limited by CloudFormation — if CFN can't do it, CDK can't either
Synthesis step — adds complexity to CI/CD pipelines
Learning curve — must know both the programming language AND AWS resource model
Stack size limits — inherits CloudFormation's 500-resource limit per stack
Breaking changes — CDK library updates can break existing constructs (semver issues)
AWS-only — synthesizes to CloudFormation, so no multi-cloud

CDK Construct Levels

Level	Description	Example
L1	Direct CloudFormation mapping (CfnBucket)	1:1 with CFN, verbose
L2	AWS-aware with sensible defaults (Bucket)	Encryption enabled by default, fewer params
L3	Opinionated patterns (ApplicationLoadBalancedFargateService)	Full architecture in one construct

Best For

Developer-heavy teams familiar with TypeScript/Python
Organizations wanting reusable infrastructure libraries (shared constructs)
Complex deployments needing loops, conditionals, and dynamic generation
Teams already using CloudFormation wanting better developer experience
Projects requiring infrastructure unit testing

Terraform: The Multi-Cloud Standard

What it is: HashiCorp's open-source IaC tool using HCL (HashiCorp Configuration Language). Declarative, provider-based architecture supporting 4,000+ providers across all major clouds and SaaS services.

Strengths

Multi-cloud — same workflow for AWS, Azure, GCP, Kubernetes, Datadog, PagerDuty, GitHub, etc.
Mature ecosystem — Terraform Registry with 15,000+ modules
HCL is purpose-built — cleaner than YAML, simpler than full programming languages
Plan before apply — terraform plan shows exactly what will change
Module system — reusable, versioned, composable modules
Import existing resources — bring manually-created resources under management
Fast execution — parallel resource creation, faster than CloudFormation for large deployments
Community — massive community, extensive documentation, Stack Overflow coverage

Weaknesses

State management — YOU manage the state file (S3 + DynamoDB locking is standard)
State drift — if someone changes resources outside Terraform, state diverges
No built-in rollback — failed applies can leave infrastructure in partial state
Provider lag — new AWS features may take days/weeks to appear in the AWS provider
License change — BSL license since 2023 (OpenTofu is the open-source fork)
HCL limitations — no full programming language features (workarounds needed for complex logic)
Sensitive data in state — state file contains secrets (must encrypt S3 backend)

Terraform vs OpenTofu

Since HashiCorp's BSL license change in 2023, OpenTofu exists as a fully open-source fork:

Criteria	Terraform	OpenTofu
License	BSL (Business Source License)	MPL 2.0 (truly open source)
Feature parity	Leading edge	Follows (slight lag)
Provider support	Full	Full (same providers)
Enterprise support	Terraform Cloud/Enterprise	Community + vendors
When to choose	Need Terraform Cloud features	Need fully open-source

Best For

Multi-cloud or hybrid environments
Platform teams managing infrastructure across multiple providers
Organizations with existing Terraform expertise
Projects needing to manage non-AWS resources (GitHub repos, DNS, monitoring, SaaS configs)
Large-scale deployments where CloudFormation is too slow

Head-to-Head Comparison

Criteria	CloudFormation	CDK	Terraform
Language	JSON/YAML	TypeScript/Python/Java/Go	HCL
Paradigm	Declarative	Imperative (synthesizes to declarative)	Declarative
Multi-cloud	❌ AWS only	❌ AWS only	✅ 4000+ providers
State management	AWS-managed	AWS-managed (via CFN)	Self-managed (S3 + DynamoDB)
Rollback	✅ Automatic	✅ Automatic (via CFN)	❌ Manual
New AWS features	Same-day	Same-day (via L1)	Days/weeks lag
Reusability	Nested stacks (limited)	Constructs (excellent)	Modules (excellent)
Testing	❌ No native support	✅ Unit + integration	✅ Terratest, plan assertions
Learning curve	Low (if you know YAML)	Medium (language + AWS)	Low-Medium (HCL)
IDE support	Basic YAML validation	Full (autocomplete, types)	Good (HCL extension)
Drift detection	✅ Built-in	✅ Built-in (via CFN)	`terraform plan` (manual)
Import existing	✅ (limited)	✅ (limited)	✅ (excellent)
Community modules	Limited	Construct Hub (growing)	Registry (massive, 15K+)
CI/CD integration	AWS-native (CodePipeline)	CDK Pipelines	Any CI tool
Cost	Free	Free	Free (OSS) / Paid (Cloud)

State Management: The Critical Difference

CloudFormation / CDK: No State File Worries

AWS tracks state internally. You never see a state file. Benefits:

No S3 backend to configure
No locking mechanism needed
No state corruption risk
Drift detection built-in

Terraform: You Own the State

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/network/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

State risks you must manage:

State file contains secrets (encrypt the S3 bucket, restrict access)
Concurrent applies can corrupt state (DynamoDB locking solves this)
Lost state = Terraform doesn't know what it manages (backup state files)
State drift = someone changed resources outside Terraform (regular terraform plan detects this)

Module / Construct Ecosystem

Terraform Registry

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  cidr = "10.0.0.0/16"
  azs  = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

15,000+ community modules. The terraform-aws-modules organization alone covers VPC, EKS, RDS, Lambda, and dozens more.

CDK Constructs

import { ApplicationLoadBalancedFargateService } from 'aws-cdk-lib/aws-ecs-patterns';

new ApplicationLoadBalancedFargateService(this, 'MyApp', {
  taskImageOptions: { image: ecs.ContainerImage.fromRegistry('my-app:latest') },
  desiredCount: 3,
  cpu: 512,
  memoryLimitMiB: 1024,
});
// This single construct creates: ECS cluster, Fargate service, ALB, target group,
// security groups, IAM roles, CloudWatch log group — all with sensible defaults

Team Workflow Patterns

Pattern 1: Terraform for Platform, CDK for Applications

Platform Team → Terraform
  ├── VPCs, Transit Gateway, DNS
  ├── EKS clusters, RDS instances
  └── Shared infrastructure (S3, KMS, IAM boundaries)

Application Teams → CDK
  ├── ECS services, Lambda functions
  ├── Application-specific resources
  └── Deploy via CDK Pipelines

Why: Platform team needs multi-provider support (AWS + GitHub + Datadog). App teams benefit from CDK's developer-friendly constructs.

Pattern 2: Terraform Everywhere

All Teams → Terraform
  ├── Shared modules in internal registry
  ├── Atlantis or Terraform Cloud for PR-based workflow
  └── One language, one workflow, one state backend

Why: Consistency. One tool to learn, one CI/CD pattern, one troubleshooting approach.

Pattern 3: CDK Everywhere

All Teams → CDK (TypeScript)
  ├── Shared construct library (internal npm package)
  ├── CDK Pipelines for deployment
  └── Jest for infrastructure testing

Why: Developer-first organization where infrastructure is code written by application developers.

The Decision Flowchart

START
  │
  ├── Do you need to manage non-AWS resources?
  │     └── YES → Terraform (multi-provider)
  │
  ├── Is your team primarily developers (TypeScript/Python)?
  │     ├── YES → CDK (familiar language, constructs, testing)
  │     └── NO → Continue ↓
  │
  ├── Do you need multi-cloud portability?
  │     └── YES → Terraform
  │
  ├── Is simplicity the priority (small team, few resources)?
  │     └── YES → CloudFormation (no tools to manage)
  │
  ├── Do you need reusable infrastructure libraries?
  │     ├── Developer org → CDK (constructs)
  │     └── Ops/platform org → Terraform (modules)
  │
  └── Already using one tool successfully?
        └── YES → Stay with it (switching cost > marginal benefit)

Migration Considerations

From	To	Effort
CloudFormation → CDK	Low	CDK can import existing CFN stacks. Migrate incrementally.
CloudFormation → Terraform	Medium	Use `terraform import`. Re-create templates in HCL.
Terraform → CDK	High	No migration path. Must re-create and import.
CDK → Terraform	High	No migration path. Must re-create and import.

Key rule: Don't migrate for marginal gains. Only migrate when the current tool is actively blocking you (e.g., multi-cloud requirement, team can't hire CloudFormation skills).

What About Pulumi / CDKTF / Crossplane?

Tool	Niche
Pulumi	Like CDK but multi-cloud. Real languages + any cloud provider. Consider if you want CDK-style + multi-cloud.
CDKTF (CDK for Terraform)	Write CDK-style code that synthesizes to Terraform HCL. Best of both worlds — but adds complexity.
Crossplane	Kubernetes-native IaC. Manages cloud resources via K8s CRDs. For teams running everything on Kubernetes.

Summary

Choose...	When...
CloudFormation	AWS-only, simple deployments, no state management overhead, same-day feature support
CDK	Developer teams, need reusable constructs, want type safety and testing, AWS-only is fine
Terraform	Multi-cloud, platform teams, large module ecosystem, existing HCL expertise, non-AWS resources

The "wrong" choice isn't which tool you pick — it's switching tools every 6 months because someone read a blog post. Pick one, standardize, and build expertise. The best IaC tool is the one your team uses consistently.

Alpesh Kumbhare is an AWS Architect at Atos, specializing in AWS infrastructure automation and IaC best practices. Connect on LinkedIn.