DEV Community: Tejas Shinkar

AWS CloudFormation — Infrastructure as Code, Templates, Stacks & Change Sets

Tejas Shinkar — Mon, 10 Aug 2026 05:46:29 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. CloudFormation is the foundation of infrastructure automation on AWS — the point where "clicking in the console" becomes "code that can be reviewed, versioned, and reproduced."

📋 Topics Covered

#	Topic	Type
1	What is IaC and Why It Exists	Concept + Interview
2	IaC Benefits	Concept
3	What is CloudFormation	Concept + Interview
4	How CloudFormation Works in the Backend	Concept + Interview
5	CloudFormation Template Structure — All Sections with Limits	Concept + Lab
6	Template Section Plain-English Explanations	Concept
7	Template vs Stack — The Distinction	Concept + Interview
8	CloudFormation Building Blocks	Concept + Cert
9	YAML Basics for CF Templates	Concept + Lab
10	CF Parameters	Concept + Lab
11	CF Rules	Concept + Cert
12	CF Mappings	Concept + Cert
13	CF Outputs	Concept + Lab
14	CF Conditions	Concept + Cert
15	CF Intrinsic Functions	Concept + Cert
16	CF Service Roles	Concept + Interview
17	CF Stack Policies and Deletion Policies	Concept + Interview
18	CF Change Sets	Concept + Lab
19	CF Stack Sets	Concept + DevOps
20	Lab — Full Build-Up with Real Failures	Lab
21	Interview Questions	Interview
22	Assignment	Practice

What is IaC and Why It Exists

Before IaC, infrastructure was managed manually — someone SSH'd into servers, clicked through console UIs, and hoped they remembered what they changed. When that person left, the knowledge went with them. When you needed three identical environments, you clicked through the console three times and hoped for no mistakes.

IaC (Infrastructure as Code) is the practice of defining and managing infrastructure through code or configuration files — so infrastructure can be created, changed, versioned, and reproduced automatically.

Instead of clicking "create EC2 instance" in the console, you write a file that says "I want an EC2 instance with these specs." You give that file to a tool. The tool creates the instance. The file goes into Git. Next time you need the same instance, you run the same file. Exact same infrastructure, zero manual steps.

The same principles that apply to application code now apply to infrastructure:

Version control — track every change with a commit message
Code review — a colleague reviews infrastructure changes before they're applied
CI/CD pipelines — infrastructure changes tested and deployed automatically
Reproducibility — the same template produces identical infrastructure every time
Documentation — the template is the documentation

IaC Benefits

Benefit	What it means in practice
Reproducibility	Same template in any region or account → identical infrastructure
Version control	Every change tracked in Git — who changed what, when, and why
Auditability	Full history of infrastructure changes — essential for compliance
Automation	CI/CD pipelines deploy infrastructure changes without human intervention
Drift detection	Compare actual infrastructure against the template — catch unauthorized manual changes
Disaster recovery	Recreate entire infrastructure in a new region in minutes
Consistency	Dev, staging, prod built from the same template — no more "works on staging, broken in prod"

What is CloudFormation

AWS CloudFormation is AWS's native IaC service. You write a template (YAML or JSON), upload it, and CloudFormation creates, updates, and manages all resources on your behalf.

Think of CloudFormation as a construction manager working from a blueprint. Your YAML/JSON template is the blueprint — it describes exactly how your AWS infrastructure should look. CloudFormation reads the blueprint, builds the infrastructure, and remembers what it built. When you change the blueprint, the construction manager doesn't demolish the entire building — they identify what changed and modify only the affected parts.

Why CloudFormation over clicking in the console:

Console clicks are not reproducible — the exact sequence is never recorded
Templates can be reviewed, approved, and audited
Same template creates identical infrastructure across dev/staging/prod
Changes can be previewed before applying via Change Sets
Automatic rollback if something goes wrong during deployment

How CloudFormation Works in the Backend

Understanding this makes troubleshooting CloudFormation failures significantly easier. This is the exact flow from class — represented as a numbered sequence.

The entire backend flow:

[1] Parse + Validate: Reads your YAML/JSON, checks for syntax errors, validates that resource types and properties are correct.

[2] Transform Expansion: Expands any transforms — SAM shorthand, Includes, Language Extensions, Modules — into full CloudFormation resource definitions.

[3] Pre-deployment Validation: Checks whether the requested resources and configuration make sense before deployment begins. Validates IAM permissions and quota limits.

[4] Build Dependency Graph (DAG): Determines resource dependencies — what depends on what and what needs to happen first. Independent resources are flagged for parallel execution. Dependent resources are ordered correctly.

[5] Compute Change Set: Compares the current AWS state (stored in CloudFormation's transactional state store) with the new template to determine exactly what needs to change — Add, Modify, Remove, Replace.

[6] Evaluate Hooks / Guardrails: Checks predefined organizational/security rules before making changes. Example: verifying every S3 bucket has encryption enabled before creation proceeds.

[7] Parallel Execution (by DAG topological order): Independent resources are created/updated simultaneously. Dependent resources follow the required order. For each resource: Resource Provider handles the actual Create/Update/Delete API call. Then → Stabilization Loop — CloudFormation polls until the resource reaches its required ready state. A successful API call doesn't mean the resource is ready. Stabilization can take 20–90 minutes for some resources (RDS, ElasticSearch, etc.).

[8] Success or Failure:

Success → State Committed: Everything reaches the desired state → stack becomes CREATE_COMPLETE or UPDATE_COMPLETE

Failure → Rollback: CloudFormation attempts to undo the changes in reverse DAG order → returns the stack toward its previous stable state

🎯 Why this matters for troubleshooting: When a stack gets stuck at CREATE_IN_PROGRESS, it's in the Stabilization Loop — waiting for a resource to become available. Check the Events tab — it shows which resource is waiting and why. When it fails, the error comes from the underlying AWS API call in the Resource Provider step.

CloudFormation Template Structure — All Sections with Limits

A CloudFormation template is a YAML or JSON file with the following structure. Every section has limits you should know for production and for the SAA-C03 exam.

AWSTemplateFormatVersion: "2010-09-09"   # optional, but always write it
Description: "..."                        # max 1024 bytes
Metadata: {}                              # console UI hints, cfn-init config
Parameters: {}                            # max 200
Rules: {}                                 # UNDER-USED — param validation logic
Mappings: {}                              # max 200 mappings, 200 attrs each
Conditions: {}                            # max 512
Transform: []                             # macro/SAM expansion
Resources: {}                             # REQUIRED — max 500
Outputs: {}                               # max 200

Only Resources is required. Every other section is optional.

Template Section Plain-English Explanations

From class — the clearest way to remember what each section does:

Section	Plain-English Question It Answers
`AWSTemplateFormatVersion`	What template format?
`Description`	What is this blueprint for?
`Metadata`	Extra blueprint information
`Parameters`	What choices should the person deploying provide?
`Rules`	Are those choices valid?
`Mappings`	Lookup table for predefined values
`Conditions`	Should this part be built or not?
`Transform`	Expand/process the blueprint
`Resources`	⭐ What actually needs to be built?
`Outputs`	What useful information should I give back after construction?

Template vs Stack — The Distinction

Template: A YAML or JSON file — the blueprint. A text file. By itself, it creates nothing. Lives in Git, S3, or locally.

Stack: What CloudFormation creates when it executes a template. A group of AWS resources managed together as one unit. The stack tracks every resource it created, their state, and the template that generated them.

The same template creates multiple stacks:

webapp.yaml (one file in Git)
→ Stack: webapp-dev → dev resources in dev account
→ Stack: webapp-staging → staging resources in staging account
→ Stack: webapp-prod → prod resources in prod account

Three environments, identical configuration, zero manual effort.

CloudFormation Building Blocks

SAM (Serverless Application Model)

Shorthand syntax for serverless — Lambda, API Gateway, DynamoDB — written in a simplified form. The Transform: AWS::Serverless-2016-10-31 macro expands SAM into full CloudFormation at deploy time. Instead of 50 lines to define a Lambda with its role and log group, SAM lets you write 10.

Include

Pulls content from another template file into the current one — splits large templates into focused, maintainable files.

Language Extensions

Adds extra syntax features — native string operations, length functions, ToJsonString — reducing repetitive code and making templates more flexible.

Modules

Reusable infrastructure components used across multiple templates. Define a "standard VPC setup" once as a module — reference it in every team's template without repeating 200 lines.

YAML Basics for CF Templates

# Comment in YAML

key: value                  # String
number: 42                  # Integer
boolean: true               # Boolean
list:                       # List/Array
  - item1
  - item2
nested:                     # Nested object/map
  key1: value1
  key2: value2
multiline: |                # Multi-line string (preserves newlines)
  line one
  line two

Indentation is everything in YAML. Unlike JSON (curly braces), YAML uses 2-space indentation to show nesting. One wrong indent fails the entire template.

CF Parameters

Parameters let you avoid hardcoding values — instead the person deploying provides them at stack creation or update time.

The same template creates a t3.micro in dev and a c5.2xlarge in prod — just by passing different values. No template changes needed.

Parameters:
  InstanceType:
    Type: String
    Default: t3.micro
    AllowedValues:
      - t3.micro
      - t3.small
      - t3.medium
      - t3.large
    Description: "EC2 instance type for the web server"

  Environment:
    Type: String
    AllowedValues:
      - dev
      - staging
      - prod
    Description: "Deployment environment"

Parameter types:

Type	Use for
`String`	Text values, names, identifiers
`Number`	Numeric values
`CommaDelimitedList`	Multiple strings
`AWS::EC2::KeyPair::KeyName`	Validates the key pair exists before deploy
`AWS::EC2::VPC::Id`	Validates the VPC ID exists before deploy
`AWS::EC2::Subnet::Id`	Validates the subnet ID exists before deploy
`AWS::SSM::Parameter::Value<String>`	Pulls value directly from SSM Parameter Store

🎯 Interview tip: AWS-specific parameter types (like AWS::EC2::VPC::Id) are validated before the stack starts deploying — an invalid VPC ID gets rejected immediately, not halfway through.

Limit: max 200 parameters per template.

CF Rules

Rules are the validation layer for Parameters — they ensure parameter combinations are valid, going beyond what AllowedValues alone can check.

AllowedValues can restrict a single parameter to a fixed list. But what if you need: "when Environment is prod, InstanceType must not be t3.micro"? That cross-parameter logic is what Rules handle.

Rules:
  ProdRequiresLargeInstance:
    Assertions:
      - Assert:
          !Or
            - !Not [!Equals [!Ref Environment, prod]]
            - !Not [!Equals [!Ref InstanceType, t3.micro]]
        AssertDescription: "Production environment cannot use t3.micro"

🎯 Note from class: Rules are marked as "UNDER-USED" in practice — most teams use Parameters with AllowedValues for simple validation and rely on Conditions for conditional resource creation. But Rules are the correct tool when you need to validate that multiple parameter values make sense together.

CF Mappings

Mappings are lookup tables baked into your template. The classic use case — AMI IDs per region, so the person deploying doesn't need to know them.

Mappings:
  RegionAMIMap:
    ap-south-1:
      AMI: ami-0abcdef1234567890
    us-east-1:
      AMI: ami-0987654321fedcba
    eu-west-1:
      AMI: ami-0a1b2c3d4e5f67890

Resources:
  MyEC2:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !FindInMap [RegionAMIMap, !Ref AWS::Region, AMI]

When deployed in Mumbai → uses Mumbai AMI automatically. The deployer doesn't touch AMI IDs.

Other uses: environment-specific instance sizes (prod gets t3.large, dev gets t3.micro), region-specific endpoint URLs.

Limit: max 200 mappings, 200 attributes each.

CF Outputs

Outputs are values CloudFormation surfaces after a stack is created — important resource identifiers made easy to find and reuse.

Outputs:
  BucketName:
    Description: "Name of the created S3 bucket"
    Value: !Ref MyBucket

  BucketARN:
    Description: "ARN of the S3 bucket"
    Value: !GetAtt MyBucket.Arn
    Export:
      Name: !Sub "${AWS::StackName}-BucketARN"

Three uses:

Console visibility — after stack creation, Outputs appear in CloudFormation console. Easy to find the ALB DNS name or RDS endpoint without digging through service consoles.
Cross-stack references — one stack exports an Output, another imports it with Fn::ImportValue. Networking stack exports VPC ID → all application stacks import it.
Automation — scripts query Outputs to get resource identifiers dynamically.

Limit: max 200 outputs per template.

CF Conditions

Conditions add boolean logic — create resources or set property values only when certain conditions are true.

Parameters:
  Environment:
    Type: String
    AllowedValues: [dev, prod]

Conditions:
  IsProduction: !Equals [!Ref Environment, prod]

Resources:
  MyBucket:
    Type: AWS::S3::Bucket
    Properties:
      VersioningConfiguration:
        Status: !If [IsProduction, Enabled, Suspended]

  ReadReplica:
    Type: AWS::RDS::DBInstance
    Condition: IsProduction    # Only created when IsProduction is true
    Properties:
      # read replica config...

Condition functions:

Function	What it does
`!Equals [a, b]`	True if a equals b
`!Not [condition]`	Inverts a condition
`!And [c1, c2]`	True if both are true
`!Or [c1, c2]`	True if either is true
`!If [condition, true_value, false_value]`	Conditional value

Limit: max 512 conditions per template.

CF Intrinsic Functions

Built-in functions that build dynamic values, reference resources, and manipulate strings.

Function	Short form	What it does
`Ref`	`!Ref`	Default identifier of a resource or parameter value
`Fn::GetAtt`	`!GetAtt`	Specific attribute of a resource (ARN, DNS name, IP)
`Fn::Sub`	`!Sub`	String substitution — inserts variable values
`Fn::Join`	`!Join`	Joins a list with a delimiter
`Fn::Select`	`!Select`	Returns one item from a list by index
`Fn::FindInMap`	`!FindInMap`	Looks up a value in a Mappings table
`Fn::If`	`!If`	Returns one of two values based on a Condition
`Fn::ImportValue`	—	Imports an Output exported by another stack
`Fn::Base64`	`!Base64`	Encodes string as Base64 (used for EC2 User Data)
`Fn::Split`	`!Split`	Splits a string into a list

!Ref vs !GetAtt — the key distinction:

!Ref MyBucket → bucket name (the default identifier for S3)
!GetAtt MyBucket.Arn → bucket ARN (a specific attribute)
!GetAtt MyInstance.PublicIp → public IP of EC2 (a specific attribute)

Practical example:

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceTypeParam          # value from Parameter
      ImageId: !FindInMap [AMIMap, !Ref AWS::Region, AMI]  # from Mappings
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-webserver"  # string substitution
      UserData:
        !Base64 |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          systemctl start httpd

Outputs:
  PublicIP:
    Value: !GetAtt MyInstance.PublicIp
  FullDNS:
    Value: !Join [".", ["api", !Ref Environment, "tejascloud.com"]]

CF Service Roles

By default, CloudFormation uses your own IAM permissions to create resources. A Service Role is an IAM Role that CloudFormation assumes instead — decoupling what humans can do from what CloudFormation can do.

Scenario: A developer needs to deploy a stack that creates VPCs, security groups, and EC2 instances. You could give the developer those IAM permissions directly — but then they could also manually create those resources outside CloudFormation, bypassing governance.

With a Service Role: The developer has cloudformation:CreateStack and iam:PassRole only. The Service Role has ec2:*, vpc:*, s3:* — whatever the stack needs. CloudFormation assumes the role and creates resources. The developer gets the infrastructure without direct access to the underlying services.

🎯 Security benefit: Least privilege for humans, necessary permissions for automation. Humans can't bypass the CloudFormation process to create resources manually.

CF Stack Policies and Deletion Policies

Stack Policy

A JSON document attached to a stack that controls which resources can be updated or replaced during a stack update.

{
  "Statement": [{
    "Effect": "Deny",
    "Action": "Update:Replace",
    "Principal": "*",
    "Resource": "LogicalResourceId/ProductionDatabase"
  }]
}

This blocks Replace operations on the ProductionDatabase resource — even if a template change would normally require it. CloudFormation fails the update rather than replace the database.

Deletion Policy

Set on individual resources inside the template — controls what happens when the stack is deleted or the resource is removed from the template.

Deletion Policy	What happens when stack is deleted
`Delete` (default)	Resource is deleted
`Retain`	Resource stays, CloudFormation stops managing it
`Snapshot`	AWS takes a backup before deleting (RDS, EBS, Redshift)

Resources:
  MyDatabase:
    Type: AWS::RDS::DBInstance
    DeletionPolicy: Snapshot    # backup before delete
    Properties: {}

  MyBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain      # keep the bucket, just stop managing it
    Properties: {}

🎯 Production rule: Always DeletionPolicy: Snapshot on RDS. Always DeletionPolicy: Retain on S3 buckets with important data. Losing a production database because of the default Delete policy is a real and painful mistake.

CF Change Sets

A Change Set is a preview of what CloudFormation will do before it actually does it — shows every resource action: Add, Modify, Remove, Replace.

A Replace action means the resource is deleted and recreated — downtime for databases, EC2 instances, etc. A Change Set lets you catch this before it happens.

The Change Set workflow:

Modify your YAML template → Upload new template → Create Change Set → CloudFormation computes the diff → Review every proposed change → If correct: Execute → CloudFormation applies → If risky or wrong: Delete the Change Set → nothing applied, stack untouched.

When to always use Change Sets: Before updating any production stack — catching a Replace before it runs is the difference between planned maintenance and an unplanned outage.

CF Stack Sets

Deploys the same CloudFormation stack across multiple AWS accounts and/or regions in a single operation.

Without Stack Sets: log into 10 accounts, switch to 3 regions in each, deploy 30 times manually.
With Stack Sets: define target accounts and regions once → CloudFormation deploys to all 30 in parallel.

Common uses: security baseline (CloudTrail, Config, GuardDuty) to every account, centralized logging in every region, standard networking across all accounts.

Two modes:

Mode	How targets are defined
Self-managed	You explicitly list account IDs and regions
Service-managed (AWS Organizations)	Auto-deploys to all accounts in an OU — new accounts added to the OU automatically get the stack

Stack Instance = one deployed stack in one specific account + region combination.

🧪 Lab — Full Build-Up with Real Failures

This is the complete lab sequence from class — building a full networking + EC2 stack resource by resource using Change Sets. The failures are included because they're part of what makes CloudFormation real — and knowing how to fix them is the skill.

Overall Lab Flow

YAML Blueprint → Parameters → VPC → Subnet → Internet Gateway → Gateway Attachment → Route Table → Security Group → Key Pair Parameter → EC2 → Dependencies via !Ref → Change Sets → Execute → Failures → Fix Template/Parameter → New Change Set → Execute → Final Infrastructure ✅

Step-by-Step Lab — Exact Sequence

Start: Create VPC + CIDR

Create Stack → initial template with just VPC and CIDR → CREATE_COMPLETE ✅

Add Subnet via Change Set

Modify template — add Subnet resource referencing VPC via !Ref → Create Change Set → Execute Change Set
❌ FAILED: Invalid Availability Zone — AZ name in template was wrong
Fix AZ in template → Create New Change Set → Execute Change Set
Subnet Created ✅

Add Internet Gateway

Modify template — add AWS::EC2::InternetGateway, add AWS::EC2::VPCGatewayAttachment using !Ref VPC and !Ref IGW → Create Change Set → Execute
MyInternetGateway Created ✅
AttachGateway Created ✅
Internet Gateway Attached to VPC ✅

Add Route Table

Modify template — add AWS::EC2::RouteTable, add Route with destination 0.0.0.0/0 → IGW, add Subnet Route Table Association → Create Change Set → Execute
Route Table Created + Associated ✅

Add Security Group

Modify template — add AWS::EC2::SecurityGroup with inbound rules (SSH 22, HTTP 80) → Create Change Set → Execute
Security Group Created ✅

Add Key Pair as Parameter + Create EC2

Modify template — add KeyPair Parameter (Type: AWS::EC2::KeyPair::KeyName), add EC2 instance referencing AMI from Mappings, SG from !Ref, subnet from !Ref, key pair from !Ref KeyPairParam → Create Change Set → Execute
❌ FAILED: Invalid/Non-existent Key Pair — wrong key pair name entered
Correct Key Pair Parameter → Create New Change Set → Execute Change Set
EC2 Created ✅

What the Lab Teaches

These failures are not bugs in the process — they're the lesson:

Invalid AZ failure: Template contained a hardcoded AZ that doesn't exist in the target region. Fix: use a Parameter for AZ name, or use !Select [0, !GetAZs !Ref AWS::Region] to dynamically pick the first available AZ.

Invalid Key Pair failure: AWS-specific parameter types (AWS::EC2::KeyPair::KeyName) validate that the key pair exists in the account — if you type a name that doesn't exist, it fails at parameter validation before any resource is created. Fix: enter the exact name of a key pair that exists in your account.

The Change Set → Execute → Fail → Fix → New Change Set → Execute cycle is normal production CloudFormation workflow. The key is that failures don't leave your infrastructure in an unknown state — CloudFormation's rollback mechanism returns you to the last stable state.

Final Template Structure (Everything Together)

AWSTemplateFormatVersion: "2010-09-09"
Description: "VPC + EC2 networking stack"

Parameters:
  KeyPairName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: "Name of an existing EC2 key pair"

  AvailabilityZone:
    Type: AWS::EC2::AvailabilityZone::Name
    Description: "AZ to deploy the subnet in"

Mappings:
  RegionAMI:
    ap-south-1:
      AMI: ami-0f5ee92e2d63afc18
    us-east-1:
      AMI: ami-0abcdef1234567890

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: "10.0.0.0/16"
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-VPC"

  MySubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref MyVPC
      CidrBlock: "10.0.1.0/24"
      AvailabilityZone: !Ref AvailabilityZone
      MapPublicIpOnLaunch: true

  MyIGW:
    Type: AWS::EC2::InternetGateway

  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref MyVPC
      InternetGatewayId: !Ref MyIGW

  MyRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC

  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: AttachGateway
    Properties:
      RouteTableId: !Ref MyRouteTable
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref MyIGW

  SubnetRouteTableAssoc:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref MySubnet
      RouteTableId: !Ref MyRouteTable

  MySecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Allow SSH and HTTP"
      VpcId: !Ref MyVPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: "0.0.0.0/0"
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: "0.0.0.0/0"

  MyEC2:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      KeyName: !Ref KeyPairName
      ImageId: !FindInMap [RegionAMI, !Ref AWS::Region, AMI]
      SubnetId: !Ref MySubnet
      SecurityGroupIds:
        - !Ref MySecurityGroup
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-WebServer"

Outputs:
  InstancePublicIP:
    Description: "Public IP of the EC2 instance"
    Value: !GetAtt MyEC2.PublicIp

  VPCID:
    Description: "VPC ID"
    Value: !Ref MyVPC

Useful Pseudo Parameters

Available automatically — no declaration needed.

Pseudo Parameter	Value
`!Ref AWS::Region`	Current region (e.g., `ap-south-1`)
`!Ref AWS::AccountId`	Current AWS account ID
`!Ref AWS::StackName`	Name of the current stack
`!Ref AWS::StackId`	Full ARN of the current stack
`!Ref AWS::NoValue`	Removes a property (used in Conditions)

⚡ Quick Revision

IaC

Define infrastructure as code → version, review, reproduce, automate
Template is the blueprint. Stack is the deployed infrastructure.
Same template → multiple stacks → identical environments

Template Structure (with limits)

AWSTemplateFormatVersion → format version, always "2010-09-09"
Metadata → UI hints, cfn-init config
Parameters → max 200, user input at deploy time
Rules → cross-parameter validation logic (under-used but important)
Mappings → max 200 mappings × 200 attrs each, lookup tables
Conditions → max 512, boolean logic for conditional resources
Transform → SAM/macro expansion
Resources → REQUIRED, max 500, the actual AWS resources
Outputs → max 200, values surfaced after stack creation

CloudFormation Backend Flow

[1] Parse + Validate → [2] Transform Expansion → [3] Pre-deployment Validation → [4] Build DAG → [5] Compute Change Set → [6] Evaluate Hooks → [7] Parallel Execution (each resource: Resource Provider → API call → Stabilization Loop 20-90 min) → [8] Success: Commit State / Failure: Rollback in reverse DAG order

Key Intrinsic Functions

!Ref → default identifier (bucket name, instance ID, parameter value)
!GetAtt → specific attribute (ARN, DNS, public IP)
!Sub → string substitution with variables
!FindInMap → lookup value in Mappings table
!If → conditional value
!Base64 → encode string for User Data

Deletion Policy

Delete (default) → resource deleted with stack
Retain → resource stays, CloudFormation stops managing
Snapshot → backup before deletion (RDS, EBS)

Change Set

Preview before applying — shows Add, Modify, Remove, Replace
Replace = resource deleted and recreated = downtime
Always use before updating production stacks

Stack Sets

Same stack → multiple accounts/regions in one operation
Service-managed mode: auto-deploys to new accounts in an OU

💼 Interview Questions

Q1: What is the difference between a CloudFormation Template and a Stack?
A template is a YAML or JSON file — the blueprint that creates nothing by itself. A stack is what CloudFormation creates when it executes a template — a group of AWS resources managed together as one unit. The same template can create multiple stacks (dev, staging, prod), each independently managed.

Q2: How does CloudFormation handle resource dependencies?
CloudFormation builds a Directed Acyclic Graph (DAG) from DependsOn attributes, !Ref references, and !GetAtt calls. Resources with no dependencies are created in parallel. Dependent resources are created in the correct order. On failure, rollback follows the reverse DAG order.

Q3: What happens during the Stabilization Loop?
After making a Create/Update/Delete API call through a Resource Provider, CloudFormation doesn't assume the resource is immediately ready. It polls the resource until it reaches its required stable state. Some resources like RDS or Elasticsearch can take 20–90 minutes to stabilize. This is why CloudFormation stacks spend a long time in CREATE_IN_PROGRESS — they're waiting for stabilization, not making slow API calls.

Q4: What is a Change Set and why should you use one before updating a production stack?
A Change Set previews every change CloudFormation will make — which resources will be Added, Modified, Removed, or Replaced. A Replace action deletes and recreates a resource, causing downtime. Using a Change Set catches this before applying, preventing unplanned outages.

Q5: What is the difference between a Stack Policy and a Deletion Policy?
A Stack Policy is a JSON document attached to a stack that prevents specific resources from being updated or replaced during stack updates — protecting critical resources from accidental replacement. A Deletion Policy is a property on individual resources inside the template that controls what happens when the stack is deleted: Delete (remove), Retain (keep), or Snapshot (backup before deleting).

Q6: What is the difference between CF Parameters and CF Rules?
Parameters define the input values a deployer provides at stack creation or update. AllowedValues can restrict a single parameter to a fixed list. Rules are validation logic that works across multiple parameters — for example, "when Environment is prod, InstanceType must not be t3.micro." Parameters control individual inputs; Rules validate that combinations of inputs make sense together.

Q7: What is !Ref vs !GetAtt?
!Ref returns the default identifier of a resource — for S3 it returns the bucket name, for EC2 it returns the instance ID. !GetAtt returns a specific named attribute — !GetAtt MyBucket.Arn returns the ARN, !GetAtt MyInstance.PublicIp returns the public IP. Use !Ref for the primary identifier, !GetAtt for everything else.

Q8: What is a CloudFormation Service Role and why would you use it?
A Service Role is an IAM Role that CloudFormation assumes to create resources, instead of using the deploying user's permissions. It enforces governance — developers can deploy CloudFormation stacks without having direct IAM permissions to create EC2s or VPCs. CloudFormation assumes the role and creates resources, keeping least privilege for humans while enabling necessary automation.

📝 Assignment

Create a CloudFormation stack that deploys an Auto Scaling Group with a Launch Template and hosts a web server using a User Data script. The web server should be accessible via an Application Load Balancer. All resources created and managed by a single CloudFormation template.

AWS Session 16 — AWS CloudFormation | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS Route 53 — DNS Fundamentals, Hosted Zones, Routing Policies & Resolvers

Tejas Shinkar — Sat, 08 Aug 2026 12:15:57 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. Route 53 is where networking meets the internet — how domain names reach your applications, how traffic gets distributed intelligently, and how AWS and on-premises networks resolve each other's names.

📋 Topics Covered

#	Topic	Type
1	DNS Pre-Requisites — How DNS Works	Concept
2	Complete DNS Resolution Flow	Concept + Interview
3	What is Route 53	Concept
4	Hosted Zones — Public vs Private	Concept + Lab
5	Hosted Zone ID	Concept + DevOps
6	DNS Record Types and Use Cases	Concept + Cert
7	NS and SOA Records — Auto-Created, Never Delete	Concept + Interview
8	Alias Record — AWS-Specific	Concept + Cert
9	Landing Zone — Brief Context	Concept
10	Route 53 Routing Policies — All 8	Concept + Cert
11	Route 53 Traffic Policies	Concept + DevOps
12	Route 53 Resolvers	Concept + Interview
13	Inbound vs Outbound Resolver Endpoints	Concept + Interview
14	Route 53 Forwarders	Concept + Interview
15	Split-Horizon DNS	Concept + Interview
16	Interview Questions	Interview
17	Practice Tasks	Practice

DNS Pre-Requisites — How DNS Works

DNS is the reason you type google.com instead of 142.250.195.46. Before understanding Route 53, these fundamentals must be solid.

Core Vocabulary

Term	What it means
Domain	Human-readable name — `google.com`, `tejascloud.in`
IP Address	Machine address — `54.21.11.90` — what computers actually use
DNS	The translation system — converts domain names → IP addresses
TLD	Top-Level Domain — the last part after the final dot
TTL	Time To Live — how long a DNS response is cached before re-querying
Recursive Resolver	Finds the answer for the client by querying other DNS servers, caches the result
Authoritative DNS Server	Stores the official DNS records for a domain — returns the definitive answer

Common TLDs:

.com → commercial · .org → organizations · .net → network · .in → India · .uk → United Kingdom · .edu → education · .gov → government

The Complete DNS Resolution Flow

This is the full journey a DNS query takes — from the moment you type a URL to the moment your browser connects to a web server.

User types tejascloud.com in browser

Step 1 — Browser Cache: Does my browser already know this IP from a recent visit? → Cache Miss → continue

Step 2 — OS Cache: Does the operating system's DNS cache have it? → Cache Miss → continue

Step 3 — Recursive Resolver (your ISP or 8.8.8.8): Does the resolver's cache have it? → Cache Miss → it begins querying on your behalf

Step 4 — Root DNS Server: "Who manages .com domains?" → returns address of the .com TLD server

Step 5 — TLD DNS Server (.com): "Who manages tejascloud.com?" → reads the NS record → returns Route 53 name server addresses

Step 6 — Authoritative DNS Server (Route 53): "What's the IP for tejascloud.com?" → checks the Hosted Zone → finds the A Record → returns 54.21.11.90

Step 7 — Recursive Resolver caches the result for the TTL duration → returns IP to the browser

Step 8 — Browser connects to 54.21.11.90 → TCP Handshake → TLS Handshake (for HTTPS) → HTTP Request → Web Server → Website loads

Why TTL matters: If your A record has TTL = 300 seconds, every DNS resolver in the world caches your IP for 5 minutes. If you change your server's IP, some users will still reach the old IP for up to 5 minutes. Before planned migrations, lower the TTL to 60 seconds a day in advance so the change propagates faster.

What is Route 53

Amazon Route 53 is AWS's fully managed, highly available, and scalable DNS service. It serves as the Authoritative DNS Server for your domains — when the TLD server (.com) asks "who manages tejascloud.com?", it's Route 53 name servers that answer.

The name "Route 53" comes from port 53 — the standard port DNS uses.

Route 53 does three things:

Domain Registration — buy and manage domain names directly through AWS
DNS Routing — host DNS records and route traffic to the correct resources
Health Checking — monitor endpoints and route traffic away from unhealthy ones

Hosted Zones — Public vs Private

A Hosted Zone is the container for all DNS records belonging to one domain. It's Route 53's way of organizing and storing the DNS configuration for tejascloud.com — all the A records, CNAME records, MX records, and so on live inside the Hosted Zone for that domain.

The library analogy:

Route 53 is the library. A Hosted Zone is one bookshelf — dedicated to a single domain. The DNS records (A, CNAME, MX, TXT) are the individual books on that shelf. When someone asks "what's the IP for tejascloud.com?" — the librarian (Route 53) goes to the right bookshelf (Hosted Zone) and finds the right book (A record).

Route 53
   │
   ├── Hosted Zone: tejascloud.com (bookshelf)
   │       ├── A Record: tejascloud.com → 54.21.11.90
   │       ├── CNAME: www → tejascloud.com
   │       ├── MX Record: mail server
   │       └── TXT Record: verification

Public Hosted Zone

Answers DNS queries coming from the public internet. Anyone anywhere can query it. This is what makes tejascloud.com accessible to the world.

Used for: public websites, customer-facing APIs, any domain that should be reachable from the internet.

Private Hosted Zone

Answers DNS queries only from within associated AWS VPCs. Invisible to the public internet. Used for internal service discovery and private naming.

Used for: internal microservices (payment.internal, db.internal), RDS endpoints with friendly names, any resource that should only be accessible within the VPC.

🎯 Interview scenario: "How do you give your internal services readable DNS names inside a VPC?" → Create a Private Hosted Zone associated with your VPC. Add A records like payment-service.internal → 10.200.1.45. EC2 instances in the VPC resolve the name automatically. The internet can't see it at all.

Hosted Zone ID

Every Hosted Zone has a unique ID (format: Z1234567890ABC). Used in automation tools to target the exact Hosted Zone when creating or updating DNS records.

In Terraform: zone_id = "Z1234567890ABC" → tells Terraform which Hosted Zone to add the record to.

In AWS CLI: aws route53 change-resource-record-sets --hosted-zone-id Z1234567890ABC ...

In CloudFormation: referenced when creating Route 53 resources programmatically.

It's not a concept you interact with in the console much — it matters most in infrastructure-as-code pipelines.

DNS Record Types and Use Cases

A DNS Record is an entry inside a Hosted Zone that tells DNS how to resolve or handle a domain name.

Record Type	Purpose	Example
A	Domain → IPv4 address	`tejascloud.com → 54.21.11.90`
AAAA	Domain → IPv6 address	`tejascloud.com → 2001:db8::1`
CNAME	Domain → Another domain	`www.tejascloud.com → tejascloud.com`
MX	Mail server for the domain	`tejascloud.com → mail.google.com`
TXT	Verification, SPF, DKIM, DMARC	`"v=spf1 include:amazon.com ~all"`
NS	Which name servers manage the domain	Auto-created by Route 53
SOA	DNS zone metadata	Auto-created by Route 53
PTR	IP → Domain (reverse DNS)	`54.21.11.90 → tejascloud.com`

Real-world example — hosting a website on EC2:

EC2 Public IP: 54.21.11.90
You create an A Record in the Hosted Zone: tejascloud.com → 54.21.11.90
User types tejascloud.com → Route 53 checks the Hosted Zone → finds the A Record → returns 54.21.11.90 → browser connects to your EC2

CNAME Limitations

CNAME cannot be used on a root domain (apex domain). You cannot create a CNAME for tejascloud.com itself — only for subdomains like www.tejascloud.com. This is a DNS standard limitation, not just Route 53. For pointing the root domain to an AWS resource, use an Alias Record instead.

NS and SOA Records — Auto-Created, Never Delete

NS Record (Name Server):

Automatically created when you create a Hosted Zone. Contains the four Route 53 name servers responsible for your domain (e.g., ns-1234.awsdns-23.com). During DNS resolution, the TLD server (.com) uses the NS record to know which Route 53 name servers manage your domain and where to send queries.

Never delete the NS record. Deleting it would break DNS resolution for your entire domain — the TLD servers would have nowhere to point queries.

SOA Record (Start of Authority):

Also automatically created. Stores DNS zone metadata — primary name server, serial number (increments on every zone change), refresh and retry intervals, expire time, and default TTL.

Never delete or modify the SOA record unless you know exactly why. It's used internally by DNS infrastructure to manage zone transfers and cache behavior.

Alias Record — AWS-Specific

The Alias Record is Route 53's solution to two problems: the CNAME root domain limitation and the need to point domains to AWS resources that don't have fixed IP addresses.

What Alias records can point to:

Application Load Balancers (ALB)
CloudFront distributions
S3 static websites
API Gateway endpoints
Elastic Beanstalk environments
Another Route 53 record in the same Hosted Zone

Why you need Alias instead of CNAME for AWS resources:

An ALB doesn't have a fixed IP — it has a DNS name like my-alb-123456789.ap-south-1.elb.amazonaws.com. And you want tejascloud.com (the root domain) to point to it. A CNAME can't be used on a root domain. An Alias record can. AWS resolves the Alias internally to the ALB's current IP, always fresh — you never manage IPs.

Key difference from CNAME:

	CNAME	Alias
Works on root domain?	❌ No	✅ Yes
Points to	Any domain	Only supported AWS resources
Health check support	No	Yes (Route 53 can health check Alias targets)
DNS query charges	Normal query charges	Free when pointing to AWS resources
Returns to client	Another domain (CNAME chain)	Actual IP directly

🎯 Cert and interview tip: "How do you point your root domain example.com to an ALB?" → Use an Alias record (not a CNAME — CNAMEs can't be on root domains). Alias is AWS-specific and resolves directly to the resource's current IPs.

Landing Zone — Brief Context

You'll hear this term in enterprise AWS contexts. A Landing Zone is a secure, pre-built AWS foundation for an entire organization — not a single account but a structured multi-account setup.

It's built using AWS Control Tower and includes: AWS Organizations (for multiple accounts), Organizational Units (OUs), IAM Identity Center (SSO), CloudTrail for audit logging, security guardrails, centralized networking, and standardized logging.

The goal: before any workload is deployed, the organization has a secure, governed, scalable starting point. Route 53 is part of this — centralized DNS for the entire organization's accounts.

This is more relevant at the architecture and governance level than day-to-day engineering — worth knowing the term and what it represents.

Route 53 Routing Policies — All 8

A Routing Policy decides which DNS answer Route 53 returns when a client queries for a domain. Route 53 supports 8 policies, each solving a different problem.

1. Simple Routing

Returns one or a few IP addresses without any logic — basic DNS. No health checks, no traffic splitting, no awareness of location or latency.

Use for: a single resource (one EC2, one ALB) with no need for intelligent routing.

Example: tejascloud.com → ALB DNS name — every user everywhere gets the same answer.

2. Weighted Routing

Splits traffic across multiple resources according to specified weights. Route 53 sends a percentage of DNS responses to each endpoint based on the weight assigned.

Example:

Weight 90 → Production ALB (version 1.0)
Weight 10 → Staging ALB (version 2.0)
Result: 90% of users hit the old version, 10% hit the new one — canary deployment.

Use for: A/B testing, canary deployments, gradually shifting traffic to a new version.

Weight calculation: Route 53 uses the ratio — if you have weights 90 and 10, it returns the first record to 90/(90+10) = 90% of queries.

3. Latency-Based Routing

Routes users to the AWS Region that provides the lowest network latency for them, based on AWS's measured latency data between the user's location and AWS Regions.

Example:

A user in India → Route 53 measures latency to Mumbai (ap-south-1) vs N. Virginia (us-east-1) → returns Mumbai record because it's closest.

Use for: any application with users in multiple geographic locations where latency matters.

🎯 Important distinction: Latency routing is based on actual measured network latency — not geographic distance. A user in Sri Lanka might have lower latency to Singapore than to Mumbai depending on network paths. Route 53 uses real data, not map distance.

4. Failover Routing

Routes traffic to a Primary resource normally. If the primary fails a health check, Route 53 automatically returns the Secondary (backup) record instead.

Example:

Primary → Production ALB in ap-south-1 (health checked)
Secondary → Backup static S3 website showing "maintenance page"
If Primary health check fails → Route 53 returns Secondary automatically

Use for: disaster recovery, high availability with a fallback endpoint.

Requires health checks on the primary record — Route 53 needs to know when to failover.

5. Geolocation Routing

Routes traffic based on the geographic location of the user — specifically their country or continent as determined by their IP address.

Example:

Requests from India → Mumbai ALB
Requests from United States → N. Virginia ALB
Requests from EU countries → Frankfurt ALB
Default → Global ALB (for any location not explicitly mapped)

Use for: serving localized content (language, pricing, regulations), data residency compliance (EU users must stay in EU), regional product catalogs.

Always configure a Default record — without it, users from unmapped locations will get no DNS response.

🎯 Geolocation vs Latency — the key difference: Geolocation routes based on where the user is located (country/continent). Latency routes based on which region responds fastest (actual network performance). A UK user might have lower latency to US East than EU-West due to network paths — Geolocation always sends them to the EU record regardless. Latency would send them to wherever is actually faster.

6. Geoproximity Routing

Routes traffic based on geographic distance between the user and the resource, with an optional bias that expands or shrinks the effective area of a region.

The key feature is bias — a positive bias makes a region "bigger" (attracts more users from surrounding areas), a negative bias makes it "smaller."

Example:

Mumbai region with bias +50 → expands Mumbai's effective reach → attracts users from South Asia, Middle East, East Africa
Singapore region with no bias → normal reach

Use for: fine-grained geographic traffic control beyond simple country/continent boundaries, gradually shifting traffic from one region to another by adjusting biases.

Requires Route 53 Traffic Flow to configure — it's the only policy that uses the visual traffic flow editor.

🎯 Geolocation vs Geoproximity: Geolocation is rigid — "India → Mumbai, USA → Virginia." Geoproximity is flexible — "route based on distance, but adjust the effective range of each region using bias." Use Geoproximity when you need to fine-tune exactly which geographic area each endpoint serves.

7. IP-Based Routing

Routes traffic based on the source IP address (CIDR range) of the DNS query, not the geographic location.

Example:

Your corporate network uses IPs in the range 203.0.113.0/24 → you want all employees on that network routed to a private internal version of the app
Everyone else → public version

Use for: routing corporate network users differently from public internet users, routing specific ISP blocks to specific endpoints, network-specific traffic management.

Difference from Geolocation: Geolocation infers location from IP geography. IP-Based routing uses exact CIDR ranges you define — much more precise for specific known IP ranges.

8. Multivalue Answer Routing

Returns up to 8 healthy records in response to a DNS query. Clients (browsers, load balancers) can pick any one and retry with another if it fails.

Example:

tejascloud.com has 5 EC2 instances each with their own health check
Route 53 returns up to 8 healthy IPs → client picks one → if it fails, client retries with another

Use for: basic DNS-level load distribution without a dedicated load balancer, improving availability by giving clients multiple options.

Not a replacement for ALB: Multivalue answer is client-side load distribution via DNS. ALB provides sophisticated server-side load balancing with health checks, connection draining, SSL termination, and more. Multivalue is a lightweight fallback — not a production load balancing strategy on its own.

Route 53 Routing Policies — Comparison Table

Policy	Routes based on	Health check support	Use for
Simple	Nothing — just returns the record	Optional	Single resource, basic DNS
Weighted	Percentage split	Yes	A/B testing, canary deployments
Latency	Measured network latency	Yes	Global apps, latency-sensitive
Failover	Primary/Secondary health	Yes (required)	DR, high availability
Geolocation	User's country/continent	Yes	Localization, compliance
Geoproximity	Geographic distance + bias	Yes	Fine-grained geographic control
IP-Based	User's source IP CIDR	Yes	Known IP range routing
Multivalue	None — returns multiple healthy IPs	Yes	Basic DNS-level distribution

Route 53 Traffic Policies

A Traffic Policy is an advanced Route 53 feature that lets you build complex routing configurations visually — combining multiple routing policies, health checks, and endpoints into one reusable policy.

For example: Use Latency routing at the top level to send users to their nearest region, then within each region use Weighted routing to split between a canary (10%) and production (90%) endpoint. This nested, multi-level routing is what Traffic Policies enable.

Traffic Policies are version-controlled — you can create a new version without affecting the live configuration, then apply it when ready. Useful for large organizations with complex, global routing needs.

Route 53 Resolvers

Route 53 Resolver is the DNS resolver built into every AWS VPC. By default, it answers DNS queries from EC2 instances for:

Public domains (via the internet)
Private Hosted Zones associated with the VPC
AWS service endpoints

The Resolver is available at the VPC's base IP + 2 address (e.g., if your VPC CIDR is 10.200.0.0/16, the resolver is at 10.200.0.2).

The challenge in hybrid environments:

When you have both an AWS VPC and an on-premises network connected via VPN or Direct Connect, two separate DNS worlds exist:

AWS Route 53 knows about AWS resources (payment.internal, RDS endpoints)
On-premises DNS knows about on-premises resources (server.company.local, printer.office.local)

Neither side can resolve the other's names by default. Resolver Endpoints bridge this gap.

Inbound vs Outbound Resolver Endpoints

Inbound Resolver Endpoint

Allows DNS queries coming from outside AWS (on-premises network) to enter the VPC and be resolved by Route 53.

Flow:

On-premises server needs to resolve app.internal (an AWS Private Hosted Zone record) → On-premises DNS server forwards the query to the Inbound Endpoint IP → Inbound Endpoint accepts it → forwards to Route 53 Resolver → Route 53 checks Private Hosted Zone → returns 10.200.1.45 → on-premises server connects to the private AWS resource

Key point: The Inbound Endpoint gives on-premises networks a target IP they can send DNS queries to — an ENI inside your VPC that acts as the entry door for external DNS queries.

Outbound Resolver Endpoint

Allows DNS queries originating inside the VPC to leave AWS and reach an on-premises DNS server.

Flow:

EC2 instance needs to resolve server.company.local (managed by on-premises DNS) → EC2 queries Route 53 Resolver → Resolver checks Forwarding Rules → matches *.company.local → Outbound Endpoint forwards the query to the on-premises DNS server IP → on-premises DNS returns the IP → Resolver returns it to EC2

Key point: The Outbound Endpoint is the exit door for DNS queries that need to leave AWS and reach on-premises DNS servers. It's always paired with Forwarding Rules that define which domain names should be forwarded (and to which on-premises DNS server IP).

Route 53 Forwarders

A Forwarding Rule (Resolver Rule) is what tells Route 53 Resolver which DNS queries to forward through the Outbound Endpoint instead of resolving locally.

Forwarding Rule: *.company.local → forward to 192.168.10.10 (on-premises DNS server IP) via the Outbound Endpoint

Any VPC query for a .company.local domain → Route 53 Resolver sees the rule → forwards to the on-premises DNS server through the Outbound Endpoint → gets the answer → returns to the EC2 instance

Types of Resolver Rules:

Rule Type	What it does
Forwarding Rule	Forward matching DNS queries to a specified IP (usually on-premises DNS)
System Rule	Auto-created by AWS — handles Private Hosted Zones and AWS-internal domains
Recursive Rule	Default — queries not matching any rule are resolved normally via internet DNS

Sharing Resolver Rules: Rules can be shared across multiple VPCs using AWS Resource Access Manager (RAM) — so a single set of forwarding rules can be applied to an entire organization's VPCs centrally.

Split-Horizon DNS

One domain, two different DNS answers depending on who's asking — without the client ever knowing.

The concept:

app.company.com — the same domain name returns different answers based on where the DNS query originates.

Query from inside the AWS VPC → Private Hosted Zone answers → returns 10.0.1.50 (private IP, internal traffic stays inside the network)

Query from the public internet → Public Hosted Zone answers → returns 54.20.30.40 (public IP, routed through the internet)

How it works in Route 53:

Create a Private Hosted Zone for app.company.com associated with your VPC → add an A record pointing to the private IP.
Create a Public Hosted Zone for app.company.com → add an A record pointing to the public IP.

When an EC2 instance inside the VPC queries app.company.com → Route 53 Resolver checks the Private Hosted Zone first → returns the private IP.

When an internet user queries app.company.com → hits the public DNS infrastructure → Public Hosted Zone answers → returns the public IP.

Why Private Hosted Zone wins for VPC queries:

The Private Hosted Zone always overrides the Public Hosted Zone for queries originating inside the associated VPC. This is by design — internal traffic should stay internal, even if a public record also exists for the same domain.

Why this pattern is valuable:

Internal users access the application over the private network — lower latency, no egress costs, more secure
External users still reach the same application via its public endpoint
The application doesn't need to change — the DNS layer handles the routing transparently

🎯 Interview definition: Split-Horizon DNS = same domain name + different DNS answer based on where the query originates. Public zone for internet users, Private zone for VPC users, private zone always wins for internal queries.

⚡ Quick Revision

DNS Resolution Flow
Browser Cache → OS Cache → Recursive Resolver → Root DNS → TLD DNS → Authoritative DNS (Route 53) → IP → Browser connects

Route 53 Basics

Authoritative DNS service for AWS-managed domains
Port 53 = standard DNS port
Does: Domain Registration + DNS Routing + Health Checking

Hosted Zones

Public Hosted Zone: internet-facing DNS, answers from anywhere
Private Hosted Zone: VPC-only DNS, invisible to internet, overrides public for VPC queries
One Hosted Zone per domain, contains all DNS records for that domain

Record Types

A → IPv4, AAAA → IPv6, CNAME → another domain (not root), MX → mail, TXT → verification/email security, NS → name servers (auto-created, never delete), SOA → zone metadata (auto-created, never delete), PTR → reverse DNS

Alias Record

AWS-specific, works on root domain (CNAME can't), points to ALB/CloudFront/S3/API GW, free DNS queries for AWS resource targets

Routing Policies

Policy	Routes based on
Simple	Nothing — just returns record
Weighted	Percentage split
Latency	Measured network latency (not distance)
Failover	Primary/Secondary health status
Geolocation	User's country/continent
Geoproximity	Distance + adjustable bias
IP-Based	Source IP CIDR range
Multivalue	Returns multiple healthy IPs

Resolvers

Inbound Endpoint: on-premises → AWS (resolve AWS Private Hosted Zone records from outside)
Outbound Endpoint: AWS → on-premises (resolve on-premises domain names from VPC)
Forwarding Rules: define which domains get forwarded through Outbound Endpoint

Split-Horizon DNS

Same domain, different answer based on query origin
Private Hosted Zone (VPC queries) → private IP
Public Hosted Zone (internet queries) → public IP
Private always wins for VPC-originated queries

💼 Interview Questions

Q1: Walk through the complete DNS resolution flow.
User types domain in browser → browser cache miss → OS cache miss → Recursive Resolver checked → if cache miss, Resolver queries Root DNS Server ("who manages .com?") → Root returns TLD server address → Recursive Resolver queries TLD server (.com) → TLD returns Route 53 name servers from the NS record → Resolver queries Route 53 (authoritative) → Route 53 checks the Hosted Zone → returns the A record (IP) → Resolver caches the result for the TTL duration → browser receives IP → TCP/TLS handshake → HTTP request → website loads.

Q2: What is the difference between a Public and Private Hosted Zone?
A Public Hosted Zone answers DNS queries from the public internet — it's how your domain is reachable by anyone. A Private Hosted Zone answers queries only from within associated AWS VPCs — it's invisible to the internet and used for internal service discovery. When a query originates from inside the VPC, the Private Hosted Zone always overrides the Public one for the same domain.

Q3: Why can't you use a CNAME for a root domain and what do you use instead?
CNAME is a DNS standard limitation — it cannot be placed on the apex (root) domain like example.com, only on subdomains like www.example.com. For pointing a root domain to an AWS resource (ALB, CloudFront, S3), Route 53 provides Alias records — an AWS-specific extension that works on root domains, supports health checks, and doesn't charge for DNS queries to AWS targets.

Q4: What is Split-Horizon DNS and how is it implemented in Route 53?
Split-Horizon DNS returns different DNS answers for the same domain depending on where the query originates. In Route 53, you create a Private Hosted Zone associated with your VPC (returns private IP for internal users) and a Public Hosted Zone (returns public IP for internet users) for the same domain. VPC queries always resolve via the Private Hosted Zone — the Private Hosted Zone overrides the Public for any query originating inside the associated VPC.

Q5: What is the difference between Route 53 Inbound and Outbound Resolver Endpoints?
An Inbound Endpoint allows DNS queries coming from outside AWS (typically an on-premises network) to enter the VPC and be resolved by Route 53 — so on-premises servers can resolve Private Hosted Zone names. An Outbound Endpoint allows DNS queries originating inside the VPC to leave AWS and reach an on-premises DNS server — so EC2 instances can resolve on-premises domain names. Outbound Endpoints work with Forwarding Rules that specify which domain patterns to forward and to which DNS server IP.

Q6: What is the difference between Geolocation and Latency-Based routing?
Geolocation routing directs traffic based on the user's geographic location (country or continent) — a UK user always goes to the EU endpoint regardless of which is actually faster. Latency-based routing directs traffic based on measured network latency — the user goes to whichever AWS Region responds fastest for them. A UK user might have lower latency to US East than EU West depending on network conditions — Latency routing would send them there; Geolocation would not.

Q7: What is Failover routing and when would you use it?
Failover routing designates one record as Primary and another as Secondary. Route 53 monitors the Primary endpoint via health checks. As long as the primary is healthy, all traffic goes there. If the health check fails, Route 53 automatically returns the Secondary record. Used for disaster recovery setups — for example, Primary is a production ALB, Secondary is an S3 static page showing a maintenance message.

Q8: What is the difference between Weighted and Multivalue routing?
Weighted routing sends specific percentages of traffic to specific endpoints — controlled, intentional splits like 90% to production and 10% to canary. Multivalue routing returns up to 8 healthy IP addresses in response to a query, and the client picks one — it's DNS-level distribution with health filtering, not traffic splitting by percentage. Weighted is for deliberate traffic control; Multivalue is for basic resilience by giving clients multiple options.

🔬 Practice Tasks

Hosted Zone lab: Register or transfer a domain to Route 53 (or use an existing one). Create a Public Hosted Zone. Add an A record pointing to an EC2 public IP. Verify DNS resolution works from your browser. Change the IP and observe how long TTL takes to propagate.
Private Hosted Zone: Create a Private Hosted Zone for internal.local associated with your VPC. Add an A record payment.internal.local → 10.200.1.45. SSH into an EC2 in the VPC and run nslookup payment.internal.local — verify it resolves to the private IP.
Split-Horizon: Create both a Public and Private Hosted Zone for the same domain. Add different A records in each. From within the VPC, verify the Private Hosted Zone answer. From your home internet, verify the Public Hosted Zone answer. Confirm they're different.
Weighted routing for canary deployment: Create two A records for the same domain with weights 90 and 10 pointing to different resources. Query the domain repeatedly and track which endpoint is returned — verify approximately 10% of responses point to the new resource.
Failover routing: Create a Primary record pointing to a running EC2 with a health check. Create a Secondary record pointing to a different endpoint. Manually stop the primary EC2. Watch the health check fail in Route 53. Verify that queries now return the Secondary record.
Resolver Endpoints (conceptual exercise): Draw the complete flow for: (a) an on-premises server resolving app.internal (an AWS Private Hosted Zone), and (b) an EC2 instance resolving server.company.local (an on-premises domain). For each, identify which endpoint (Inbound or Outbound) is used and in which direction the DNS query travels.

AWS Session 15 — Amazon Route 53 | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS SQS & SNS — Decoupling Services with Queues, Topics & Fan-Out Patterns

Tejas Shinkar — Sat, 08 Aug 2026 01:37:58 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session covers how distributed systems communicate reliably — the backbone of every production-grade AWS architecture.

📋 Topics Covered

#	Topic	Type
1	Monolithic vs Microservices Architecture	Concept + Interview
2	Why Messaging Services Exist	Concept
3	Amazon SQS — What It Is	Concept + Interview
4	SQS — Pull-Based vs Push-Based	Concept + Interview
5	SQS Configuration Settings	Concept + Lab
6	Visibility Timeout — Deep Dive	Concept + Interview
7	SQS Message Lifecycle — With and Without Failure	Concept + DevOps
8	Key SQS APIs	Concept + Lab
9	Dead Letter Queue (DLQ)	Concept + Interview
10	SQS Standard vs FIFO Queue	Concept + Cert
11	SQS FIFO Deduplication	Concept + Cert
12	Two Core SQS Integration Patterns	Concept + DevOps
13	Amazon SNS — What It Is	Concept + Interview
14	SNS vs SQS — Push vs Pull	Concept + Interview
15	SNS Security and Message Filtering	Concept + Cert
16	SNS + SQS Fan-Out Pattern	Concept + DevOps
17	Amazon SNS FIFO	Concept + Cert
18	SNS FIFO + SQS FIFO Fan-Out	Concept + Cert
19	Production Architecture View	DevOps
20	Lab — Deploy SNS and SQS Queues	Lab
21	Interview Questions	Interview
22	Assignment	Practice

Monolithic vs Microservices Architecture

Before queues and topics make sense, you need to understand the problem they solve.

Monolithic Architecture

The entire application — UI, business logic, database layer, authentication, notifications, payment — is packaged and deployed as a single unit.

Early advantages: simple to develop, test, and deploy — everything is in one place.

Why it breaks down at scale:

A bug in the notification module can crash the entire application — including payments
Scaling one feature (payments under high load) requires scaling the entire application
Deploying any small change means redeploying everything — risk and coordination overhead
The codebase grows into something nobody fully understands anymore

Microservices Architecture

The application is divided into small, independently deployable services, each responsible for exactly one business capability.

What this looks like: Order Service → Payment Service → Notification Service → Inventory Service → User Service — each deployed separately, scaled independently, owned by its own team.

Advantages: fault isolation (Notifications failing doesn't affect Payments), independent scaling, each service uses the best technology for its job, teams deploy without coordinating with each other.

The challenge microservices introduce:

If Payment needs to tell Notification "payment completed, send email" — what happens if Notification is temporarily down? Does Payment fail too? This tight synchronous coupling defeats the purpose of separation.

This is exactly the problem SQS and SNS solve.

🎯 Interview tip: When asked "why microservices?" — cover independent deployment, fault isolation, team autonomy, and technology flexibility. When asked "what challenges come with microservices?" — inter-service communication, distributed tracing, and data consistency are the honest answers.

Why Messaging Services Exist

In a direct synchronous call: Order Service → HTTP → Notification Service → if Notification is down, Order fails too. Tight coupling.

With a queue: Order Service → puts message in queue → returns success immediately → Notification reads from queue when ready.

The queue acts as a buffer and decoupler — the producer doesn't care when the consumer processes, just that it eventually does. Traffic spikes are absorbed by the queue instead of crashing downstream services.

Amazon SQS — What It Is

Amazon SQS (Simple Queue Service) is a fully managed, distributed message queue that temporarily stores messages until a consumer is ready to process them.

The bank token system analogy:

You walk into a bank. You don't go directly to the cashier — you take a numbered token. The token machine doesn't solve your problem. It organizes the waiting line so cashiers can handle customers at a manageable pace. Customers (producers) keep arriving. Cashiers (consumers) process at their own pace. SQS is the token machine.

Key facts:

Fully managed — no servers, clusters, or infrastructure to provision
Automatically distributed across multiple backend partitions
Replicated across multiple AZs — messages are durable even if one AZ fails
Scales to billions of messages automatically
Once a message is consumed and deleted, it's gone

SQS — Pull-Based vs Push-Based

SQS is pull-based. Consumers actively poll the queue using ReceiveMessage() when they're ready. SQS never pushes messages.

Think of it like checking your email inbox — you open it when you're ready. Your inbox doesn't force your browser open when mail arrives.

Why pull-based is powerful: consumers control their own processing rate, each consumer scales independently based on queue depth, if a consumer goes down messages safely queue up until it recovers.

Long Polling vs Short Polling:

Short Polling (Wait Time = 0, default): Asks "any messages?" → SQS responds immediately, even if empty → you pay for the API call even when nothing was there.

Long Polling (Wait Time up to 20s): Asks "any messages?" → SQS waits up to 20 seconds → if a message arrives, it returns immediately → if nothing arrives, it returns at timeout. Fewer API calls, lower cost, faster response when messages exist.

🎯 Production tip: Always use Long Polling (20 seconds). It reduces empty responses, lowers costs, and responds faster when messages actually arrive.

SQS Configuration Settings

Configuration	One-Line Meaning
Visibility Timeout	Hides a message while one consumer is processing it — prevents double-processing
Message Retention Period	How long an unprocessed message stays before being auto-deleted (1 min – 14 days, default 4 days)
Delivery Delay	How long after sending before consumers can see the message (0 – 15 minutes)
Maximum Message Size	Max size per message — 256 KB
Receive Message Wait Time	How long ReceiveMessage waits when queue is empty — enables Long Polling (0 – 20 seconds)

Visibility Timeout — Deep Dive

The mechanism that prevents two consumers from processing the same message simultaneously.

How it works:

Consumer A calls ReceiveMessage() → SQS returns Message X → SQS hides Message X from all other consumers for the Visibility Timeout duration → Consumer A processes → Consumer A calls DeleteMessage() → Message permanently gone.

If Consumer A crashes or takes too long → Visibility Timeout expires → Message reappears → Consumer B picks it up and retries.

The ChangeMessageVisibility pattern for long-running jobs:

Consumer picks the message → starts processing → periodically calls ChangeMessageVisibility() to extend the timeout before it expires → processing completes → DeleteMessage().

🎯 Interview tip: It's the consumer itself (Lambda/EC2/ECS) that calls ChangeMessageVisibility — not a separate monitoring process. The consumer extends its own timeout while processing, then deletes the message on success.

Setting the right Visibility Timeout: If your job typically takes 45 seconds, set the timeout to at least 90 seconds — a 2× buffer accounts for occasional slowness without triggering false retries.

SQS Message Lifecycle — With and Without Failure

Without failure (happy path):

Customer → Producer → SendMessage() → SQS Queue → Consumer calls ReceiveMessage() → Visibility Timeout starts → Consumer processes (DB write, email, payment) → Consumer calls DeleteMessage() → Done ✅

With failure (retry path):

Customer → Producer → SQS → Consumer picks message → Processing fails (crash, exception) → Visibility Timeout expires → Message reappears → Another consumer retries → Success → DeleteMessage() ✅

If it keeps failing after maxReceiveCount attempts → Message moves to the Dead Letter Queue

Key SQS APIs

API	What it does
`SendMessage()`	Put a message into the queue
`ReceiveMessage()`	Read messages from the queue (up to 10 at a time)
`ChangeMessageVisibility()`	Extend the Visibility Timeout for a message being processed
`DeleteMessage()`	Remove a successfully processed message permanently
`PurgeQueue()`	Delete all messages in the queue at once
`DeleteQueue()`	Remove the entire queue

Dead Letter Queue (DLQ)

Some messages genuinely cannot be processed — corrupt data, an unrecoverable bug, an unexpected format. Without a DLQ, these messages retry forever, consuming consumer capacity and blocking healthy messages.

A Dead Letter Queue is a separate SQS queue that receives messages which have failed after exceeding the configured maxReceiveCount (maximum retry attempts).

Also called the poisoned pill defense — a message that would kill your consumer over and over gets quarantined instead of contaminating normal flow.

The flow:

Producer → SQS Queue → Consumer → Failure → Retry → Retry → Retry (maxReceiveCount reached) → Message moved to DLQ → Normal queue continues processing healthy messages

What to do with DLQ messages:

Inspect the failed message — understand what was in it
Find the root cause — bug in consumer? Bad data from producer?
Fix the application or the message data
Use Redrive to replay the message back to the source queue after the fix is deployed

🎯 Interview Q: Does DLQ automatically fix failed messages? → No. DLQ only stores them safely. A human (or automation) must investigate, fix the root cause, and replay the message.

Production DLQ best practices:

Always configure a DLQ on every production queue — without one, failed messages silently disappear after retention expires
Set DLQ retention to 14 days (maximum) — maximum time to investigate
Set CloudWatch alarm on DLQ's ApproximateNumberOfMessagesVisible metric — get paged the moment any message lands there
After fixing root cause, use Redrive to replay — don't manually reprocess

maxReceiveCount: Typical production value is 3–5 retries. Too low = transient errors moved to DLQ prematurely. Too high = a bad message wastes consumer resources for too long.

SQS Standard vs FIFO Queue

	Standard Queue	FIFO Queue
Ordering	Best-effort (may arrive out of order)	Strict first-in, first-out — guaranteed
Delivery	At-least-once (duplicates possible)	Exactly-once (within 5-minute deduplication window)
Throughput	Nearly unlimited	300 messages/second (3,000 with batching)
Deduplication	Not supported	Built-in
Queue name	Any name	Must end in `.fifo`
Use case	Notifications, logs, parallel jobs where order doesn't matter	Financial transactions, order processing, inventory — where order and exactly-once matter

How to choose: Is message order critical to correctness? → FIFO. Can you afford occasional duplicates? → Standard. Need maximum throughput? → Standard.

SQS FIFO Deduplication

FIFO queues guarantee exactly-once processing within a 5-minute deduplication window. If the same message is sent twice within 5 minutes, the duplicate is silently discarded.

Two deduplication methods:

Content-Based Deduplication: SQS generates a SHA-256 hash of the message body. Identical hash within 5 minutes = duplicate, dropped.

Message Deduplication ID: The producer explicitly sends a unique ID with each message. Same ID within 5 minutes = duplicate, dropped. Gives producers explicit control over deduplication.

Message Group ID:

Within a FIFO queue, messages with the same Group ID are processed in strict order. Messages with different Group IDs can be processed in parallel.

Example: order processing queue where Group ID = order_id. All events for order 1001 (created → paid → shipped → delivered) are in strict order. Events for order 1002 process in parallel with order 1001's events — FIFO within groups, parallelism across groups.

Two Core SQS Integration Patterns

Pattern 1 — Producer-Consumer Decoupling

Order Service sends a message to SQS when an order is placed → returns success to user immediately → Fulfillment Service reads from the queue at its own pace → processes independently.

If Fulfillment is slow or down, orders safely queue up — no data lost, no error shown to the user. When Fulfillment recovers, it processes the backlog.

The core value of SQS: services don't need to know about each other, don't need to be available at the same time, and traffic spikes are absorbed by the queue instead of crashing downstream services.

Pattern 2 — Workflow Orchestration

For multi-step processes (Order → Payment → Warehouse → Shipping → Notification), SQS handles message passing between steps while AWS Step Functions orchestrates the overall workflow — managing state, retries, and branching logic.

SQS: the messaging between steps. Step Functions: the sequencing, state tracking, and failure handling at the workflow level.

Amazon SNS — What It Is

Amazon SNS (Simple Notification Service) is a fully managed push-based Publish/Subscribe messaging service.

The pub/sub model:

Publishers send a message to an SNS Topic — not to individual consumers. SNS immediately fans out a copy to every subscriber of that topic simultaneously. Each subscription receives its own independent copy.

SNS is not a queue. It doesn't store messages long-term — once published, SNS immediately delivers to subscribers and moves on. SNS is a broadcasting megaphone, not a holding room.

Supported subscriber types:

Subscriber	What happens
SQS queue	Message dropped into the queue for async processing
Lambda function	Function invoked immediately
Email	Message sent to the address
SMS	Text message sent
HTTP/HTTPS endpoint	POST request sent to the URL
Mobile push (APNs, GCM)	Push notification to iOS/Android

SNS vs SQS — Push vs Pull

	SQS	SNS
Model	Pull-based — consumers poll	Push-based — SNS delivers immediately
Storage	Yes — stores until consumed (up to 14 days)	No — delivers and moves on
Consumers	One message → typically one consumer	One message → all subscribers simultaneously
Use for	Decoupling, buffering, async work queues	Broadcasting, fan-out, real-time notifications

They're complementary, not competing. SQS for reliable async processing; SNS for immediate fan-out broadcasting.

SNS Security and Message Filtering

Security

Encryption in transit: HTTPS by default on all SNS API calls and deliveries.

Encryption at rest: SSE via AWS KMS — encrypts message content before delivery.

Access control:

SNS Topic Policy: Resource-based policy on the topic — defines who can publish and subscribe. Used for cross-account access (similar to S3 bucket policies).
IAM Policy: Controls which IAM users/roles can call SNS APIs (sns:Publish, sns:Subscribe).

Message Filtering

By default, every subscriber receives every message. Filter policies let each subscription define which messages it actually receives, based on message attributes.

Example:

An Order topic receives all order events. Fulfillment only cares about status: "PLACED". Analytics wants everything. Fraud only cares about amount > 10000.

Without filtering: every service processes every event and discards irrelevant ones — wasteful.
With filtering: each subscription only receives matching messages — SNS filters before delivery.

The Fulfillment filter policy:

{ "status": ["PLACED"] }

SNS evaluates this filter before delivery. Messages where status is not "PLACED" are not delivered to the Fulfillment subscription at all.

SNS + SQS Fan-Out Pattern

One of the most important architectural patterns in AWS — appears constantly in interviews and certifications.

The problem: When an order is placed, Fulfillment, Inventory, Notifications, and Analytics all need to react. If you call each directly, you're back to tight coupling — one slow service makes the entire order placement slow.

The fan-out solution:

Producer publishes one message to an SNS Topic → SNS immediately delivers to all subscribed SQS queues simultaneously → each service has its own queue → each service processes at its own pace, independently.

Visualized:

New Order Event → SNS Topic (order-events)
→ SQS Queue → Fulfillment Service (has its own DLQ, scaling, retry logic)
→ SQS Queue → Inventory Service (has its own DLQ, scaling, retry logic)
→ SQS Queue → Notification Service (has its own DLQ)
→ SQS Queue → Analytics Service (has its own DLQ)

Why this pattern is powerful:

Publish once, SNS handles delivery to all consumers
Each consumer is fully isolated — if Analytics is slow, Fulfillment is unaffected
Adding a new consumer = add a new SQS subscription to the topic, no producer changes needed
Each queue has independent DLQ, retry logic, and scaling

🎯 Interview Q: "How would you design a system where one event triggers multiple independent services?" → SNS + SQS fan-out. One SNS topic, one SQS queue per service, each with its own consumer and DLQ.

Amazon SNS FIFO

For cases where message order matters at the broadcasting level.

	Standard SNS	SNS FIFO
Ordering	Best-effort	Strict ordering within a Message Group
Deduplication	No	Yes — same mechanism as SQS FIFO
Throughput	Very high	Up to 300 published messages/second
Subscribers	Any type	Only SQS FIFO queues

Key constraint: SNS FIFO can only fan out to SQS FIFO queues. If you need FIFO ordering, the entire delivery chain must be FIFO.

SNS FIFO + SQS FIFO Fan-Out

The most controlled and ordered fan-out pattern:

Publisher → SNS FIFO Topic → SQS FIFO Queue A (Fulfillment) → strict order guaranteed
→ SQS FIFO Queue B (Inventory) → strict order guaranteed

When to use this vs standard fan-out:

If order of events matters for correctness downstream — inventory updates where "add 10 units" then "remove 15 units" must process in exact order — use SNS FIFO + SQS FIFO.

If order doesn't matter (analytics, notifications), use standard SNS + SQS — higher throughput, simpler, cheaper.

Production Architecture View

This is what a real event-driven system looks like combining everything from this session:

User places order → Order Service → publishes to SNS Topic

SNS fans out simultaneously to:

SQS Queue (Fulfillment) → consumer: ECS service, has DLQ monitored by CloudWatch

SQS Queue (Inventory) → consumer: Lambda, has DLQ

SQS FIFO Queue (Payment Audit) → strict order for audit trail

Email subscription → sends order confirmation directly to customer

Each SQS queue uses Long Polling (20s), Visibility Timeout set to 2× expected processing time, DLQ with 14-day retention, CloudWatch alarm on DLQ depth, Redrive configured for replay after fixes.

🧪 Lab — Deploy SNS and SQS Queues

Step 1 — Create SQS Queue + DLQ

SQS Console → Create queue → Type: Standard → Name: order-processing-queue → Visibility Timeout: 30s → Message Retention: 4 days → Receive Message Wait Time: 20s → Create queue.

Create second queue: Name: order-processing-dlq → Create queue.

Go back to order-processing-queue → Edit → Dead-letter queue → Enable → Select order-processing-dlq → maxReceiveCount: 3 → Save.

Step 2 — Create SNS Topic

SNS Console → Topics → Create topic → Type: Standard → Name: order-events → Create topic.

Step 3 — Subscribe SQS to SNS

SNS topic page → Create subscription → Protocol: SQS → Endpoint: ARN of order-processing-queue → Create subscription.

Then: SQS queue → Access Policy → add the SNS topic ARN as a principal with sqs:SendMessage permission on this queue (allows SNS to write to SQS).

Step 4 — Test Fan-Out

SNS topic → Publish message → enter test payload → Publish.
Go to order-processing-queue → Send and receive messages → Poll for messages → see the message SNS delivered ✅

Step 5 — Test DLQ

Receive a message from the queue 3 times without deleting it (simulating failure) → Check order-processing-dlq → message should have moved there ✅

⚡ Quick Revision

Monolithic vs Microservices

Monolithic: single unit, simple initially, hard to scale and maintain
Microservices: per-capability services, independent deployment, fault isolation

SQS

Pull-based distributed queue — consumers poll, SQS never pushes
At-least-once delivery, best-effort ordering (Standard) vs exactly-once, strict FIFO (FIFO queue)
Standard: unlimited throughput. FIFO: 300/sec (3000 with batching), name ends in .fifo
Long Polling (20s): always use in production

SQS Key Config

Visibility Timeout: hides message during processing, prevents double-processing
DLQ: receives messages after maxReceiveCount failures, does NOT auto-fix, set 14-day retention, monitor with CloudWatch alarm
ChangeMessageVisibility: consumer extends its own timeout for long jobs

SQS API Flow
SendMessage → ReceiveMessage → ChangeMessageVisibility (if needed) → DeleteMessage (on success)

SNS

Push-based Pub/Sub — one publish → immediate delivery to all subscribers simultaneously
Not a queue — no long-term storage
Subscribers: SQS, Lambda, Email, SMS, HTTP, Mobile Push
Message Filtering: per-subscription JSON filter policy on message attributes

Fan-Out (SNS + SQS)

One SNS topic → multiple SQS queues → each consumer independent
Best pattern for: one event → multiple independent services
Add a new consumer: just add a new SQS subscription — no producer changes

FIFO

SNS FIFO: strict ordering, deduplication, only delivers to SQS FIFO queues
Use SNS FIFO + SQS FIFO when order correctness matters end-to-end
Use Standard SNS + SQS when order doesn't matter — higher throughput, cheaper

💼 Interview Questions

Q1: What is the difference between SQS and SNS?
SQS is pull-based — consumers poll for messages, messages are stored until consumed (up to 14 days), and typically one consumer processes each message. SNS is push-based pub/sub — publishers send to a topic, SNS immediately pushes to all subscribers simultaneously, and messages aren't stored long-term. They're complementary: SNS for fan-out broadcasting, SQS for reliable async processing.

Q2: What is the Visibility Timeout in SQS and why is it important?
When a consumer reads a message, it becomes invisible to all other consumers for the Visibility Timeout duration. This prevents double-processing — only one consumer handles the message at a time. If the consumer succeeds and deletes the message, it's gone. If the consumer fails, the timeout expires, the message reappears, and another consumer can retry.

Q3: What is a Dead Letter Queue and when would a message end up there?
A DLQ is a separate SQS queue that receives messages which have failed processing after exceeding maxReceiveCount. A message lands there when a consumer repeatedly can't process it — due to a bug, corrupt data, or an unrecoverable error. The DLQ doesn't fix anything — it safely stores failed messages for investigation, root cause analysis, and optional replay via Redrive after the issue is fixed.

Q4: What is the difference between SQS Standard and SQS FIFO?
Standard offers at-least-once delivery (duplicates possible) and best-effort ordering with nearly unlimited throughput. FIFO guarantees exactly-once delivery (5-minute deduplication window) and strict ordering, but is limited to 300 messages/second (3,000 with batching). Use Standard when order and deduplication don't matter. Use FIFO for financial transactions, order processing, inventory — where sequence is critical.

Q5: Explain the SNS + SQS Fan-Out pattern and why it's useful.
A producer publishes one message to an SNS topic. SNS immediately delivers a copy to all subscribed SQS queues simultaneously. Each downstream service has its own queue, consumer, DLQ, and retry logic — completely independent. This means one event triggers multiple services without tight coupling. Adding a new consumer is just adding a new SQS subscription — no changes to the producer. Failures in one consumer don't affect others.

Q6: Your SQS consumer takes 10 minutes to process a message but the Visibility Timeout is 30 seconds. What happens and how do you fix it?
After 30 seconds, the timeout expires and the message reappears. Another consumer picks it up — now two consumers process the same message, risking data corruption or duplicate operations. Fix: the consumer should periodically call ChangeMessageVisibility() to extend the timeout before it expires. Alternatively, set Visibility Timeout to at least 2× the maximum expected processing time.

Q7: How does message filtering work in SNS?
Each subscription defines a filter policy — a JSON document specifying which message attribute values the subscription cares about. SNS evaluates the filter before delivery. If the message attributes don't match, SNS doesn't deliver that message to that subscription. This prevents every consumer from receiving and discarding irrelevant messages, reducing processing cost and simplifying consumer logic.

Q8: What is the difference between SNS FIFO and Standard SNS?
Standard SNS delivers with best-effort ordering at very high throughput and supports all subscriber types (SQS, Lambda, Email, SMS, HTTP). SNS FIFO guarantees strict ordering and deduplication within a Message Group, but is limited to 300 messages/second and can only deliver to SQS FIFO queues. Use SNS FIFO when the order of events matters for correctness downstream and the entire delivery chain must be ordered.

📝 Assignment

Create a CloudWatch alarm (on any metric — EC2 CPU, SQS queue depth, or a custom metric) and configure it to send notifications to a list of email addresses or aliases via an SNS topic. Verify that when the alarm triggers, all email addresses receive the notification.

AWS Session 14 — SQS & SNS Messaging | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS Lambda — Serverless Compute, Cold Starts, Invocation Models & VPC Integration

Tejas Shinkar — Fri, 07 Aug 2026 07:27:54 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session moves from managed databases into serverless compute — a fundamentally different execution model that changes how you think about infrastructure entirely.

📋 Topics Covered

#	Topic	Type
1	What is Serverless — The Mental Model	Concept
2	Lambda vs EC2 — When to Use What	Concept + Interview
3	Key Components of Lambda	Concept
4	Lambda Handler — Entry Point	Concept + Lab
5	Firecracker — What Actually Runs Lambda	Concept
6	Cold Start vs Warm Invocation	Concept + Interview
7	The Lambda Init Phase	Concept + DevOps
8	Provisioned Concurrency — Eliminating Cold Starts	Concept + Cert
9	Best Practices for Production Lambda	DevOps
10	Lambda Triggers	Concept + Lab
11	Invocation Models — Synchronous, Asynchronous, Poll-Based	Concept + Interview
12	Execution Roles and Permissions	Concept + Lab
13	Lambda in a VPC — Private Resource Access	Concept + Interview
14	Lambda Pricing	Concept
15	Lab — Lambda Triggered by S3	Lab
16	Interview Questions	Interview
17	Assignment	Practice

What is Serverless — The Mental Model

"Serverless" doesn't mean there are no servers. It means you don't see, manage, or think about servers — AWS runs them invisibly on your behalf.

With traditional compute (EC2), you provision a server, choose its size, install the OS, deploy your code, and pay for that server 24/7 whether it's handling traffic or sitting idle. Serverless inverts all of that — you write a function, upload it, and AWS runs it only when something triggers it. The infrastructure appears and disappears automatically.

The core shift: In EC2, you manage servers that run your code. In Lambda, you write code and AWS manages everything that runs it.

Three defining traits of serverless:

No server provisioning or management — AWS handles all infrastructure
Automatic scaling — from zero to thousands of concurrent executions, instantly
Pay only for what you use — billed per request + per millisecond of execution time (when idle, you pay nothing)

Lambda vs EC2 — When to Use What

The Chef Analogy

EC2 = Hire a full-time chef. The chef is at the restaurant all day, every day — whether three customers arrive or three hundred. You pay the chef's salary regardless of how many orders come in.

Lambda = Call a chef only when an order arrives. No orders → no cost. An order comes in → the chef cooks → leaves. You pay only for the cooking time.

Side-by-Side Comparison

	EC2	Lambda
Server management	You manage (patching, scaling, monitoring)	AWS manages everything
Startup model	Always running	Runs only on trigger
Max runtime	Indefinite	15 minutes per invocation
Scaling	Manual or ASG	Automatic, instant
Billing	Per hour (even when idle)	Per request + per GB-second of execution
Use for	Long-running apps, stateful services, databases	Event-driven, short-lived tasks, automation
State	Can maintain state on disk	Stateless — no persistent state between invocations

When Lambda is the right choice

Responding to S3 file uploads (resize an image, parse a CSV)
Processing messages from SQS, SNS, or Kinesis
API backends triggered by API Gateway
Scheduled tasks (run a report every night at midnight via EventBridge)
Real-time stream processing (DynamoDB Streams, Kinesis)
Glue code between services in an event-driven pipeline

When Lambda is NOT the right choice

Long-running processes (more than 15 minutes — use EC2 or Fargate)
Applications that hold state in memory across requests
Workloads requiring persistent local disk (use EC2 with EBS)
High-performance compute (HPC, ML training — use GPU EC2)

🎯 Interview Q: Can Lambda run forever? → No. Maximum execution timeout is 15 minutes. If your task takes longer, it must be broken into smaller steps (Step Functions) or moved to EC2/Fargate.

Key Components of Lambda

Component	What it is
Function	Your code — packaged and deployed to Lambda
Runtime	The language environment (Python 3.12, Node.js 20, Java 21, Go, etc.)
Handler	The specific function inside your code that Lambda calls on each invocation
Layers	Reusable packages (libraries, dependencies) shared across multiple functions
Execution Role	An IAM Role that grants Lambda permission to call other AWS services
Trigger	The event source that invokes the function (S3, API Gateway, SQS, etc.)
Log Streams	CloudWatch Log Streams where function output and errors are captured
Function Settings	Memory (128 MB to 10 GB), timeout (up to 15 min), environment variables, concurrency limits

Lambda Handler — The Entry Point

The handler is the entry point to your Lambda function — the specific function in your code that AWS calls when the function is triggered. You define it in the Lambda console as filename.function_name.

Example in Python:

def lambda_handler(event, context):
    print("Event received:", event)
    return {
        "statusCode": 200,
        "body": "Hello from Lambda!"
    }

The two parameters every handler receives:

event: A dictionary/object containing all the information about what triggered this invocation — for an S3 trigger, it contains the bucket name and object key; for API Gateway, it contains the HTTP method, path, and request body; for SQS, it contains the message contents.

context: An object with metadata about the invocation itself — the function name, the remaining execution time, the request ID, the CloudWatch log stream name.

The return value of the handler becomes the response — for a synchronous invocation like API Gateway, the response is returned to the caller. For asynchronous invocations, the return value is ignored.

Firecracker — What Actually Runs Lambda

When you invoke a Lambda function, something needs to execute your code in an isolated, secure environment. AWS built Firecracker specifically for this.

Firecracker is AWS's lightweight microVM (micro virtual machine) technology — it provides the security and isolation of a full virtual machine with the speed and low resource usage needed for serverless workloads.

Think of it this way: a traditional VM takes seconds to start because it boots a full OS. A Docker container starts faster but shares the host kernel (less isolation). Firecracker hits the middle — it's a minimal VM with its own kernel that starts in milliseconds, uses very little memory, and is fully isolated from all other Lambda functions running on the same physical server.

Why Firecracker matters:

Lambda functions from different customers can run on the same physical hardware with full isolation between them — security without sacrificing density
Cold starts happen in milliseconds instead of seconds because Firecracker microVMs boot extremely fast
The same technology powers AWS Fargate (serverless containers)

Cold Start vs Warm Invocation

This is one of the most important Lambda concepts for both interviews and production use.

Cold Start

A cold start happens when Lambda needs to create a brand new execution environment to run your function — because no existing one is available (the function hasn't been invoked recently, or concurrency is ramping up faster than existing environments can handle).

What happens during a cold start:

AWS allocates a Firecracker microVM → loads the Lambda runtime (Python, Node, etc.) → downloads and unpacks your function code → runs all code outside the handler (the Init Phase) → then finally runs the handler.

This entire process adds latency — typically 100ms to a few seconds depending on the runtime, package size, and how much initialization code runs.

Warm Invocation

A warm invocation happens when Lambda reuses an existing execution environment that's already fully initialized from a previous invocation.

AWS already has the Firecracker microVM running, the runtime loaded, and your code in memory → it directly invokes the handler → skips the entire Init Phase.

This is significantly faster — no initialization overhead, just the handler execution time.

The Lifecycle Visualized

First invocation (cold):
Create microVM → Load runtime → Download code → Run Init Phase (code outside handler) → Run handler → Return response → Environment stays warm for a while

Second invocation (warm):
Reuse existing environment → Run handler directly → Return response

After extended idle period:
AWS recycles the environment → Next invocation is cold again

🎯 Interview Q: After how much idle time does a Lambda become cold again? → AWS doesn't guarantee a fixed idle timeout — the execution environment may be recycled at any time after being idle. The timing isn't published and isn't guaranteed. If consistent low latency is critical, use Provisioned Concurrency.

The Lambda Init Phase

The Init Phase occurs only during a cold start. It's the window between "Lambda decided to run your function" and "Lambda actually calls your handler."

What runs during Init Phase:

Everything outside the handler function — connection setup, SDK initialization, loading configuration, reading secrets from Secrets Manager or Parameter Store.

Example — good production pattern:

import boto3
import os

# This runs ONCE during Init Phase (cold start only)
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ['TABLE_NAME'])
secret = get_secret()  # loaded once, reused on warm invocations

def lambda_handler(event, context):
    # This runs on EVERY invocation
    result = table.get_item(Key={'id': event['id']})
    return result

Why this matters in production:

If you initialize your DynamoDB client inside the handler, it gets created on every single invocation — wasting time and compute. If you initialize it outside (at module level), it's created once during the cold start and reused on all subsequent warm invocations — dramatically reducing per-invocation latency and cost.

🎯 Best practice: Always initialize reusable resources (DB connections, SDK clients, loaded configuration, decrypted secrets) outside the handler. They survive across warm invocations. Inside the handler, only put code that must run per-invocation.

Provisioned Concurrency — Eliminating Cold Starts

For latency-sensitive production workloads (APIs where users are waiting for a response), cold starts are unacceptable. Provisioned Concurrency is the solution.

Provisioned Concurrency keeps a specified number of Lambda execution environments pre-initialized and ready — the Firecracker microVM is running, the runtime is loaded, and your Init Phase code has already executed. When a request arrives, it immediately hits a warm environment, with zero cold start latency.

Important distinction:

Provisioned Concurrency does NOT send dummy requests to keep the function warm — it physically keeps initialized execution environments alive in a ready state. There's no artificial traffic, just pre-warmed infrastructure.

The cost trade-off:

You pay for provisioned concurrency even when no requests are coming in — you're paying to keep those environments alive. This is a deliberate trade-off: spend slightly more to guarantee consistent low latency for your users.

When to use it:

Customer-facing APIs where p99 latency matters
Payment or checkout flows where slow responses lose sales
Any Lambda function behind API Gateway serving real users

When NOT to use it:

Background processing (users aren't waiting)
Dev/test environments
Functions with predictable low traffic

Best Practices for Production Lambda

These come directly from what you learned in class, framed as actionable rules:

1. Keep your deployment package small. Every MB of package size adds cold start latency because Lambda must download and unpack it. Use Lambda Layers for shared dependencies and avoid bundling unused libraries.

2. Initialize reusable resources outside the handler. Database connections, SDK clients, secrets — put them at module level so they're created once (Init Phase) and reused across warm invocations.

3. Set timeouts deliberately. Don't leave the default timeout at 3 seconds blindly — understand your function's expected runtime and set a meaningful timeout. Too short = unexpected failures. Too long = runaway functions costing money.

4. Keep logging meaningful. Every log line your function emits goes to CloudWatch and is charged. In production, avoid debug-level logging on every invocation — log only errors, warnings, and meaningful business events.

5. Use environment variables for configuration. Never hardcode secrets, endpoints, or table names in your code. Use environment variables (or Secrets Manager for sensitive values) so the same code works across dev/staging/prod environments.

6. Limit function scope. One Lambda function should do one thing. A function that tries to do too much becomes hard to test, debug, and maintain.

Lambda Triggers

A trigger is an event source that invokes your Lambda function. Lambda functions don't run on their own — they always respond to something.

Trigger	What causes it	Common use
S3	File uploaded, deleted, or modified in a bucket	Image processing, ETL, notifications
API Gateway	HTTP request (GET, POST, PUT, DELETE)	REST APIs, webhooks
DynamoDB Streams	Item inserted, updated, or deleted in a DynamoDB table	Real-time data sync, audit logging
SQS	Message added to a queue	Decoupled background processing
SNS	Message published to a topic	Fan-out notifications, email, SMS
EventBridge	Scheduled rule or custom event	Cron jobs, automation pipelines
Kinesis	Records added to a data stream	Real-time analytics, log processing
ALB	HTTP request via Application Load Balancer	Serverless web applications
Cognito	User pool events (signup, login)	Custom auth flows, post-signup triggers

🎯 Interview Q: What triggers a Lambda function? → Events from AWS services — S3 uploads, API Gateway requests, DynamoDB Streams, SQS messages, SNS notifications, EventBridge schedules, and more. Lambda is fundamentally event-driven.

Invocation Models — Synchronous, Asynchronous, Poll-Based

Not all Lambda triggers work the same way — the invocation model determines who calls Lambda, who waits for the response, and who handles retries on failure.

The Three Models

	Synchronous	Asynchronous	Poll-Based
Examples	API Gateway, ALB, SDK direct call	S3, SNS, EventBridge	SQS, Kinesis, Kafka, DynamoDB Streams
Who triggers Lambda?	Caller directly	AWS service (push)	Lambda (polls the source)
Caller waits?	Yes — waits for response	No — fire and forget	N/A
Retry on failure	Caller's responsibility	Lambda retries automatically (up to 2 times)	Event source + Lambda polling logic
Failure destination	Error returned to caller	Dead Letter Queue or EventBridge	DLQ or failure destination

Memory Tricks

Synchronous → 📞 Phone Call
You call someone and stay on the line waiting for their answer. If they don't pick up, you know immediately — you handle the retry.

Asynchronous → 📧 Email
You send the email and move on with your day. The recipient processes it when they're ready. If delivery fails, the mail system retries, not you.

Poll-Based → 📬 Checking Your Mailbox
You walk to the mailbox every few minutes to see if anything arrived. That's exactly what Lambda does with SQS, Kinesis, Kafka, and DynamoDB Streams — Lambda continuously polls the source and processes whatever it finds.

Deep Dive Per Model

Synchronous:

The caller waits for Lambda to finish and return a response. If Lambda throws an error, the error is returned directly to the caller — the caller (application code or API Gateway) decides whether to retry.

Asynchronous:

AWS queues the event and returns a 202 Accepted to the caller immediately — Lambda processes it when it can. If Lambda fails, AWS automatically retries up to 2 times. After all retries are exhausted, the failed event goes to a configured Dead Letter Queue (SQS or SNS) or an EventBridge failure destination for investigation.

Poll-Based:

Lambda itself polls the event source (SQS queue, Kinesis stream, DynamoDB Stream) in a loop. When it finds records, it invokes itself with a batch of them. This model is used for queue and stream processing — Lambda reads as fast as the source produces events.

🎯 Interview tip: "How does S3 invoke Lambda?" → Asynchronously — S3 pushes an event notification to Lambda, doesn't wait for a response, and Lambda retries automatically on failure. "How does API Gateway invoke Lambda?" → Synchronously — the HTTP response the user sees depends on Lambda's return value.

Execution Roles and Permissions

Lambda functions need IAM permissions to interact with other AWS services — they can't access S3, DynamoDB, or CloudWatch by default.

Execution Role = an IAM Role attached to a Lambda function that defines what AWS services and actions the function is allowed to call.

Example: Lambda triggered by S3, writing results to DynamoDB

The Execution Role for this function needs:

s3:GetObject on the source bucket (to read the uploaded file)

dynamodb:PutItem on the target table (to write the result)

logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents on CloudWatch (to write logs)

The last three — CloudWatch log permissions — are included in the AWS managed policy AWSLambdaBasicExecutionRole, which is the minimum every Lambda function needs.

Principle of Least Privilege applies here too: Give Lambda only the permissions it actually needs, nothing more. A function that reads from S3 shouldn't also have DynamoDB write permissions unless it genuinely needs them.

🎯 Common interview scenario: "Your Lambda function is failing with an AccessDenied error when trying to write to DynamoDB. What do you check first?" → Check the Lambda function's Execution Role — verify it has dynamodb:PutItem permission on that specific table ARN.

Lambda in a VPC — Private Resource Access

The Default Setup

By default, Lambda runs inside an AWS-managed VPC — this gives it internet access for calling public AWS service endpoints or external APIs, but it cannot reach resources inside your private VPC — your RDS database, ElastiCache cluster, or private EC2 instances.

Connecting Lambda to Your VPC

To access private resources, you attach Lambda to your VPC by selecting subnets and Security Groups — similar to how you'd configure any other EC2-based resource.

Under the hood, AWS uses Hyperplane ENIs (pre-created, managed elastic network interfaces) to connect Lambda to your VPC. This is significantly faster than the older model where Lambda created a new ENI on every cold start — Hyperplane ENIs are shared and pre-warmed, which reduces cold start latency for VPC-attached functions.

The Internet Access Trade-Off

Key point: Once Lambda is attached to your VPC, it loses direct internet access. It now operates like any other private subnet resource.

This creates a dependency decision:

Lambda needs to access	Solution
Private resources (RDS, ElastiCache, internal EC2)	Attach Lambda to your VPC, select private subnets
Public internet (third-party APIs, external services)	Route traffic through a NAT Gateway in a public subnet
AWS services (S3, DynamoDB, Secrets Manager)	Use VPC Endpoints — private connectivity without internet or NAT

🎯 Interview Q: "Your Lambda function is attached to your VPC and can access your RDS database, but it can't reach an external payment API. What do you configure?" → Add a NAT Gateway to a public subnet in the same VPC. Update the private subnet's route table to route 0.0.0.0/0 through the NAT Gateway. Lambda can then reach the internet through NAT while still accessing private resources directly.

Flow with full connectivity:

Lambda (in private subnet) → needs internet (external API) → route table sends to NAT Gateway (in public subnet) → NAT Gateway → Internet Gateway → external API

Lambda (in private subnet) → needs S3 or DynamoDB → VPC Gateway/Interface Endpoint → directly to AWS service, no internet involved

Lambda Pricing

Lambda billing has two components plus one commonly overlooked cost:

Component 1 — Requests:
You're charged per million invocations. The first 1 million requests per month are free.

Component 2 — Duration (GB-seconds):
Execution time multiplied by the memory you allocated. A function with 512 MB of memory running for 2 seconds = 1 GB-second of compute.

More memory = higher cost per second, but often faster execution (Lambda allocates CPU proportionally to memory). Sometimes giving a function more memory actually reduces cost because it finishes faster.

Component 3 — CloudWatch Logs (often overlooked):

Every print() statement, every log line your Lambda function emits goes to CloudWatch Logs — and CloudWatch charges for log ingestion and storage. For functions invoked millions of times per day, excessive logging becomes a real cost.

🎯 Production discipline: In production, log errors and important business events. Remove or disable debug-level logging that fires on every invocation. The savings add up at scale.

Free Tier (permanent, not just 12 months):

1 million requests/month
400,000 GB-seconds of compute/month

🧪 Lab — Lambda Triggered by S3

What you're building: Upload an image to S3 → Lambda is triggered → (in the assignment, Lambda resizes the image and saves the thumbnail back to S3)

Step 1 — Create the Lambda Function

AWS Console → Lambda → Create function

Choose: Author from scratch
Function name: s3-image-processor
Runtime: Python 3.12
Architecture: x86_64

Execution role: Create a new role with basic Lambda permissions (adds CloudWatch Logs access automatically)
→ Create function

Step 2 — Configure S3 Trigger

In the Lambda function page → Add trigger → Select S3
Bucket: select your source bucket
Event type: PUT (triggers on file upload)
Prefix/Suffix: optionally filter to .jpg or .png files only
→ Add

Step 3 — Write the Handler

import json
import boto3

def lambda_handler(event, context):
    # Extract bucket and object key from the S3 event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    print(f"New file uploaded: s3://{bucket}/{key}")

    return {
        'statusCode': 200,
        'body': json.dumps(f'Processed: {key}')
    }

Step 4 — Add Required Permissions to Execution Role

Lambda function → Configuration → Permissions → Click the execution role link → IAM console opens
Add inline policy or attach managed policy:

s3:GetObject on the source bucket

s3:PutObject on the destination bucket (for the assignment)

Step 5 — Test

Upload any file to your S3 bucket → Go to Lambda → Monitor → View CloudWatch logs
You should see the log line: New file uploaded: s3://your-bucket/your-file.jpg

⚡ Quick Revision

Lambda Core

Serverless — no server management, auto-scales, pay per use
Max timeout: 15 minutes
Stateless — no persistent state between invocations

Cold Start vs Warm

Cold: create microVM → load runtime → download code → Init Phase → run handler (slow)
Warm: reuse existing environment → run handler directly (fast)
AWS may recycle idle environments at any time — no guaranteed idle timeout

Init Phase

Runs only on cold start
Executes all code outside the handler (DB connections, SDK clients, secrets)
Initialize reusable resources here — they survive across warm invocations

Provisioned Concurrency

Keeps N environments pre-initialized → zero cold starts for those N concurrent requests
Costs money even when idle — pay for consistent latency

Invocation Models

Model	Memory trick	Who retries?
Synchronous	📞 Phone call — wait for answer	Caller
Asynchronous	📧 Email — send and move on	Lambda (auto, up to 2x)
Poll-Based	📬 Check mailbox — Lambda polls	Event source + polling logic

VPC Integration

Default: Lambda in AWS VPC — has internet, no private resource access
VPC-attached: Lambda in your VPC — access private resources, loses internet
Fix for internet: add NAT Gateway
Fix for AWS services: use VPC Endpoints (private, no internet needed)

Firecracker

Lightweight microVM — full isolation, millisecond startup
Powers Lambda and Fargate

Pricing

Per request + per GB-second (memory × duration)
CloudWatch Logs cost is easy to overlook — keep logging lean in production

💼 Interview Questions

Q1: What is the difference between Lambda and EC2?
EC2 is a persistent virtual machine you provision, manage, and pay for continuously — whether handling traffic or idle. Lambda is serverless — it runs only when triggered, scales automatically, charges only for the execution time used, and has zero infrastructure management overhead. EC2 is for long-running stateful services; Lambda is for event-driven, short-lived tasks.

Q2: What is a cold start in Lambda and how do you reduce it?
A cold start occurs when Lambda creates a new execution environment — it must initialize the Firecracker microVM, load the runtime, download the function code, and run the Init Phase before the handler executes. This adds latency. To reduce cold starts: keep the deployment package small, initialize reusable resources outside the handler, and use Provisioned Concurrency for latency-critical functions.

Q3: What is the difference between synchronous, asynchronous, and poll-based invocation?
Synchronous invocation (API Gateway, ALB) means the caller waits for Lambda's response and handles retries itself. Asynchronous invocation (S3, SNS, EventBridge) means AWS queues the event, returns immediately to the caller, and Lambda retries automatically on failure. Poll-based invocation (SQS, Kinesis, DynamoDB Streams) means Lambda itself continuously polls the source and processes records in batches.

Q4: Why does Lambda lose internet access when attached to a VPC?
Lambda in a VPC operates like any resource in a private subnet — it routes traffic through the subnet's route table. Since private subnets don't have a direct route to the Internet Gateway, Lambda has no internet path. To restore internet access, you add a NAT Gateway in a public subnet and route the private subnet's 0.0.0.0/0 traffic through it — the same pattern used for any private EC2 instance.

Q5: What is Provisioned Concurrency and when would you use it?
Provisioned Concurrency keeps a specified number of Lambda execution environments pre-initialized — the microVM, runtime, and Init Phase code are already ready. When requests arrive, they immediately hit warm environments with zero cold start latency. You'd use it for customer-facing APIs or payment flows where consistent low latency is critical. It costs money even when idle, so it's not appropriate for background or batch processing.

Q6: What should you initialize outside the Lambda handler vs inside it?
Outside the handler (at module level, runs once during Init Phase): database connection clients, SDK clients (boto3 sessions), secrets loaded from Secrets Manager, configuration loaded from environment variables, pre-computed static data. Inside the handler (runs on every invocation): logic that depends on the event itself — parsing the event, executing the business logic, returning the response.

Q7: Your Lambda function returns AccessDenied when calling DynamoDB. What do you check?
Check the Lambda function's Execution Role in IAM — verify it has the required DynamoDB permissions (dynamodb:GetItem, dynamodb:PutItem, etc.) on the correct table ARN. The Execution Role is the IAM identity Lambda uses when calling other AWS services, and missing permissions here is the most common cause of AccessDenied errors from Lambda.

📝 Assignment

Build an image resize pipeline: when an image is uploaded to an S3 bucket, Lambda automatically resizes it to a thumbnail and saves the result to a different S3 prefix or bucket.

AWS Session 13 — Lambda Fundamentals | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS Aurora, ElastiCache Patterns & DynamoDB — The Complete Data Layer

Tejas Shinkar — Fri, 07 Aug 2026 00:54:38 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session completes the database picture — Aurora's read/write architecture, ElastiCache caching strategies, and DynamoDB from table creation to production-ready query patterns.

📋 Topics Covered

#	Topic	Type
1	Aurora Endpoints — Writer vs Reader	Concept + Interview
2	What Happens When the Aurora Writer Fails	Concept + Cert
3	ElastiCache Caching Patterns — Lazy Loading, Write Through, Session Store	Concept + Interview
4	Cache Invalidation	Concept + Interview
5	DynamoDB — What It Is and When to Use It	Concept + Interview
6	DynamoDB Table Creation — Keys and Settings	Concept + Lab
7	Table Classes — Standard vs Standard-IA	Concept + Cert
8	Capacity Modes — On-Demand vs Provisioned	Concept + Cert
9	Warm Throughput	Concept + Cert
10	DynamoDB Items & Attributes — CRUD Operations	Concept + Lab
11	Query vs Scan — The Critical Difference	Concept + Interview
12	Local Secondary Index (LSI) vs Global Secondary Index (GSI)	Concept + Cert
13	Bonus Concepts — Streams, DAX, Consistency, Transactions	Concept + Interview
14	Interview Questions	Interview
15	Practice Tasks	Practice

Aurora Endpoints — Writer vs Reader

Aurora doesn't give you just one database endpoint — it gives you two, each serving a different purpose and routing to different parts of the cluster.

Writer Endpoint (Primary Endpoint): Always points to the current primary/writer instance. All write operations (INSERT, UPDATE, DELETE) go here. If a failover happens and a replica is promoted, Aurora automatically redirects this endpoint to the new writer — your application's configuration never needs to change.

Reader Endpoint: A load-balanced endpoint that distributes read-only queries (SELECT) across all available Aurora Replicas. You don't manage which replica serves each query — Aurora handles the routing, spreading read traffic evenly across however many replicas exist.

Why this architecture matters:

In a typical application, reads far outnumber writes — a product page might be read thousands of times per second but updated once a day. Without a reader endpoint, all reads and writes compete for the same primary instance. With Aurora's reader endpoint, reads are offloaded to replicas — the writer handles only writes, and read capacity scales horizontally by adding more replicas.

🎯 Interview Q1: Why do we need a Reader Endpoint?

To distribute read-only queries across multiple Aurora Replicas. This reduces load on the writer instance, improves response times for reads, and lets you scale read capacity horizontally by adding replicas — all without any change to the application connection string.

🎯 Interview Q2: What happens if the Aurora writer fails?

Aurora automatically promotes one of the existing replicas to become the new writer. The Writer Endpoint automatically redirects to the new primary. Because the endpoint itself doesn't change (only what it resolves to), the application needs zero configuration changes. Aurora's failover is typically faster than standard RDS Multi-AZ — often under 30 seconds.

Connection to last session: This is exactly the same principle as RDS using a DNS endpoint instead of a static IP — the endpoint is a stable address that AWS reroutes behind the scenes. Your application always connects to the same string, and AWS handles where that string points.

ElastiCache Caching Patterns

ElastiCache isn't just "put stuff in cache" — how you cache data matters significantly for consistency, performance, and staleness. There are three standard patterns, each designed for a different use case.

Pattern Comparison

Pattern	How it works	Stale data?	Best for
Lazy Loading	Cache data only on the first read (cache miss triggers DB query + cache write)	Yes (possible)	Product catalog, blogs, non-critical reads
Write Through	Update cache and database together on every write	No (always in sync)	Banking, inventory, user profiles
Session Store	Store temporary session data in cache with a TTL	N/A (time-bound)	Shopping carts, login sessions, user state

Lazy Loading (Cache-Aside)

Application checks cache → Cache Hit: return data immediately → Cache Miss: query the database, write result to cache, return data to caller.

The good: Only data that's actually requested gets cached — no wasted cache memory on data nobody reads. Very simple to implement.

The trade-off: On a cache miss, the call takes longer because it hits the database AND writes to the cache. And there's a window where the cache can serve stale data if the database was updated without invalidating the cache entry.

How stale data is handled in Lazy Loading:

When a database record is updated, the corresponding cache entry is explicitly deleted (invalidated). On the next read, a cache miss occurs, the latest data is fetched from the database, and the cache is refreshed with the fresh value. A TTL (Time To Live) is often set on cache entries as a backup — even if something isn't explicitly invalidated, the cache automatically expires it after the TTL elapses.

Write Through

Every database write also updates the cache at the same time — both happen together, never independently.

The good: The cache is always synchronized with the database. No stale data ever. When an application reads from cache, it always gets the current value.

The trade-off: Every write incurs extra latency because two writes happen (DB + cache) instead of one. Cache memory may be used for data that's written often but rarely read.

🎯 When to choose which: If consistency is critical (banking, inventory, anything financial) → Write Through. If reads dominate and occasional staleness is acceptable (product listings, blog content) → Lazy Loading. If you're storing temporary state that shouldn't live in the database at all → Session Store.

Session Store

Temporary session data (login tokens, shopping cart contents, user preferences) is stored in ElastiCache with a TTL. When the TTL expires, the session data is automatically deleted.

Why cache, not the database: Session data is read and written on every request, changes frequently, and has a natural expiry — it doesn't need the durability guarantees of a relational database. Storing it in a distributed cache means any application server can retrieve any user's session without sticky sessions.

Analogy: Session store is like a coat check at a restaurant. You hand over your coat (session data) when you arrive, get a token (session ID), and the coat is held temporarily. If you don't come back within closing time (TTL), the restaurant donates it. You never permanently owned the storage — it was always meant to be temporary.

DynamoDB — What It Is and When to Use It

Amazon DynamoDB is a fully managed, serverless, key-value and document NoSQL database. Unlike RDS, there are no servers to provision, no capacity planning for OS or engine, no connection limits to manage — you define a table, put data in it, and DynamoDB scales automatically.

When to choose DynamoDB over RDS:

Choose RDS when	Choose DynamoDB when
Data has complex relationships, needs JOINs	Data access patterns are simple and predictable
Strong ACID transactions across many tables	Ultra-low latency at massive scale
Team is already SQL-native	Flexible or evolving schema (semi-structured data)
Reporting and analytics queries	Serverless, no schema migration headaches

What DynamoDB is NOT for: Complex analytical queries, ad-hoc SQL reporting, or workloads where you don't know your access patterns ahead of time. DynamoDB is optimized for known, repetitive access patterns — "give me user 1234's orders" not "give me the average order value grouped by region for Q4."

DynamoDB Table Creation — Keys and Settings

The Primary Key — Two Flavors

Every item in a DynamoDB table must have a unique primary key. You define this at table creation and cannot change it later.

Option 1 — Partition Key only (Simple Primary Key)

The partition key alone uniquely identifies every item. DynamoDB uses this key to determine which physical partition stores the item (via internal hashing). No two items can share the same partition key value.

Good for: user profiles (user_id is unique per user), product catalog (product_id is unique per product).

Option 2 — Partition Key + Sort Key (Composite Primary Key)

Two attributes together form a unique identity. Items with the same partition key are grouped together and sorted by the sort key. This allows related items to live close together physically, making range queries efficient.

Good for: orders (partition key = user_id, sort key = order_timestamp → "give me all orders for user 1234, sorted by time"), messages (partition key = chat_id, sort key = message_timestamp → "give me all messages in this chat, newest first").

🎯 Interview tip: The partition key is the access gateway — your most common query pattern must be expressible using it. If you can't answer "what's my partition key?" you haven't finished designing your table.

Table Settings — What Each Option Does

Table Class:

Class	Use case	Cost note
Standard	Frequently accessed data	Higher storage cost, optimized for read/write throughput
Standard-IA (Infrequent Access)	Data accessed rarely	Lower storage cost (~60% cheaper), higher per-read/write cost

Capacity Mode — the most important config decision:

	On-Demand	Provisioned
How it works	DynamoDB auto-scales instantly per request	You specify exact RCUs and WCUs upfront
Traffic pattern	Unpredictable, spiky, or new workloads	Predictable, steady, well-known workloads
Cost model	Pay per request (higher per-unit cost)	Pay for reserved capacity (lower cost if fully utilized)
Throttling risk	None — handles any traffic instantly	Yes — excess traffic is throttled
Typical use	Dev/test, early-stage products, variable traffic	Production workloads with known baselines

RCU and WCU — what they measure:

One Read Capacity Unit (RCU) = one strongly consistent read per second for an item up to 4 KB. One Write Capacity Unit (WCU) = one write per second for an item up to 1 KB.

The Capacity Calculator in the console estimates how many RCUs and WCUs your workload needs — you input item size, read/write rate, and consistency requirement. Eventually consistent reads cost half a WCU each.

Warm Throughput

When DynamoDB allocates capacity to a new table, it starts conservatively. If a sudden burst of traffic hits immediately — say, a product launch — DynamoDB may throttle requests until it has time to scale up internally.

Warm Throughput pre-warms the table at creation time, telling DynamoDB to allocate higher baseline capacity from the start, so that sudden traffic spikes on day one don't cause throttling.

Think of it as pre-heating an oven before putting food in — you don't wait for it to warm up slowly after the food is already in there. You configure the starting temperature so it's ready when you need it.

DynamoDB Items & Attributes — CRUD Operations

Key Terms

Table: the container for your data (like a spreadsheet)
Item: a single record in the table (like a row)
Attribute: a field on an item (like a column) — but unlike SQL, different items in the same table can have different attributes (schema-flexible)

Example — two items in the same table with different attributes:

{ "user_id": "u-1234", "name": "Tejas", "email": "tejas@example.com", "city": "Nashik" }
{ "user_id": "u-5678", "name": "Rahul", "phone": "+91-9876543210" }

No schema migration needed — DynamoDB doesn't enforce that every item has the same attributes.

CRUD in the DynamoDB Console

Create: Use "Create item" → add attribute name and value → can switch between form view and JSON view → save.

Read: Use Query or Scan (see next section) to retrieve items, or open an item directly from the results list.

Update: Select an item → edit any attribute value → save. The partition key and sort key cannot be changed — they define the item's identity.

Delete: Select an item → delete. This permanently removes it.

The JSON view is particularly useful for seeing exactly how DynamoDB stores your data internally — every attribute has a type tag (S for string, N for number, BOOL for boolean, L for list, M for map).

Query vs Scan — The Critical Difference

This is one of the most important performance decisions in DynamoDB, and it comes up in almost every DynamoDB interview question.

	Query	Scan
How it works	Retrieves items using the Partition Key (+ optional Sort Key)	Reads the entire table, then filters
Efficiency	Very efficient — reads only the relevant partition	Very inefficient — reads everything regardless
Cost	Low — charged only for data actually returned	High — charged for the entire table read
When to use	Production workloads — always prefer this	Occasional admin tasks, small tables, migrations

🎯 The rule: In production, you should almost never Scan. If you find yourself needing to Scan frequently, it's a signal that your table's primary key design doesn't match your access patterns — and you need either a redesign or an additional index (LSI/GSI).

Concrete example:

You have an Orders table with partition key = user_id and sort key = order_date.

Query: "Give me all orders for user_id = 'u-1234' placed after 2026-01-01" → DynamoDB goes directly to that partition, reads only those items. Fast and cheap.

Scan: "Give me all orders where order total > 5000" → DynamoDB reads every single order in the table across all partitions, then filters. Slow and expensive — the filter happens after reading everything.

Filters with Query: You can add filter expressions to a Query result, but filters are applied after items are read from the partition — they reduce what you see, not what DynamoDB reads and charges for. Sort key conditions, however, genuinely reduce what is read.

Local Secondary Index (LSI) vs Global Secondary Index (GSI)

Sometimes your query patterns require filtering or sorting by an attribute that isn't your primary key. Indexes let you do this efficiently without falling back to Scan.

Analogy: Indexes in DynamoDB are like the index at the back of a book — instead of reading every page to find mentions of "VPC", you flip to the index and jump directly to the relevant pages. DynamoDB maintains this alternative lookup structure automatically.

Local Secondary Index (LSI)

Same Partition Key as the base table, but a different Sort Key. Allows you to sort or filter items within the same partition using a different attribute.

Example: Orders table has partition key = user_id, sort key = order_date. You also need to query orders sorted by order_total for the same user. Create an LSI with sort key = order_total.

Constraints:

Must be created at table creation time — cannot add later
Maximum 5 LSIs per table
Shares provisioned throughput with the base table
Only queries within a single partition (same partition key as base table)

Global Secondary Index (GSI)

A completely independent index with its own Partition Key and optional Sort Key — totally different from the base table's keys.

Example: Orders table has partition key = user_id. You want "all orders with status = 'PENDING' sorted by order_date." Create a GSI with partition key = status and sort key = order_date.

Constraints:

Can be created at any time — much more flexible than LSI
Maximum 20 GSIs per table
Has its own separate provisioned throughput (additional cost)
Queries can span all partitions of the base table (truly global)

LSI vs GSI Comparison

	LSI	GSI
Partition Key	Same as base table	Different (you define it)
Sort Key	Different from base table	Optional, you define it
When created	At table creation only	Any time
Throughput	Shared with base table	Separate (extra cost)
Query scope	Single partition	Entire table
Max per table	5	20
Use when	Need different sort within same partition	Need to query by a completely different attribute

🎯 Interview trap: "Can you add an LSI after creating a DynamoDB table?" → No. LSIs must be defined at creation. If you realize you need one later, you'd have to recreate the table and migrate data. GSIs can be added any time. Always think through your access patterns before creating a DynamoDB table.

Bonus Concepts — Worth Knowing

DynamoDB Streams

DynamoDB can stream a record of every change (INSERT, UPDATE, DELETE) to items in a table — useful for event-driven architectures.

Common pattern: DynamoDB item changes → Stream captures the event → Lambda is triggered → processes the change (send a notification, update a search index, replicate to another system, populate a cache).

DAX — DynamoDB Accelerator

DynamoDB's own in-memory caching layer, built specifically for DynamoDB. Reduces read latency from milliseconds to microseconds. Unlike ElastiCache (which is general-purpose), DAX is DynamoDB-native and requires no application code changes — it's a drop-in cache using the exact same DynamoDB API.

🎯 DAX vs ElastiCache for DynamoDB: If your application already uses DynamoDB and just needs faster reads, DAX is simpler (same API, no code changes). ElastiCache gives more flexibility if you need to cache data from multiple sources.

Eventually Consistent vs Strongly Consistent Reads

DynamoDB replicates data across multiple AZs. On a write, data propagates across replicas within roughly one second.

	Eventually Consistent (default)	Strongly Consistent
Data freshness	Might be slightly stale	Always the most up-to-date
RCU cost	0.5 RCU per 4 KB	1 full RCU per 4 KB
Latency	Lower	Slightly higher
Use when	Most general reads	Must read your own write immediately

When it matters: For most use cases (product catalog, session data), eventual consistency is fine. For financial transactions or reservation systems where you must immediately read your own write, use strongly consistent reads.

DynamoDB Transactions

DynamoDB supports ACID transactions via TransactGetItems and TransactWriteItems — you can atomically read or write multiple items across multiple tables, either all succeeding or all failing together. This addresses the common misconception that NoSQL databases can't handle transactions.

⚡ Quick Revision

Aurora Endpoints

Writer Endpoint → always points to current primary → handles all writes
Reader Endpoint → load-balances reads across all replicas
On writer failure → replica promoted automatically → Writer Endpoint reroutes → zero app config changes

Caching Patterns

Pattern	Stale data?	Use for
Lazy Loading	Yes (until invalidated or TTL expires)	Read-heavy, occasional stale ok
Write Through	No	Consistency-critical (banking, inventory)
Session Store	N/A (TTL-controlled)	Login sessions, shopping carts

Cache invalidation in Lazy Loading: delete cache entry on DB update → next read repopulates.

DynamoDB Keys

Partition Key only → must be unique per item
Partition Key + Sort Key → partition groups related items, sort key orders them within a partition

Capacity Modes

On-Demand: pay per request, no throttle, higher unit cost → unpredictable workloads
Provisioned: fixed RCUs/WCUs, lower cost if utilized, throttles on excess → predictable production

Query vs Scan

Query: uses Partition Key, reads only relevant data → fast, cheap → always prefer
Scan: reads entire table → slow, expensive → admin tasks only

Indexes

LSI: same partition key, different sort key, create at table creation only, max 5
GSI: own partition key, create any time, max 20, separate throughput cost

Bonus

DAX: DynamoDB-native microsecond cache, no code changes needed
Streams: captures item-level changes for Lambda/event-driven patterns
Strongly Consistent Read: 2× the RCU cost vs eventually consistent, guaranteed fresh data
Transactions: ACID across multiple items/tables — DynamoDB does support transactions

💼 Interview Questions

Q1: What is the difference between the Writer Endpoint and the Reader Endpoint in Aurora?
The Writer Endpoint always points to the current primary instance and handles all write operations. The Reader Endpoint load-balances read queries across all Aurora Replicas. If the writer fails, Aurora promotes a replica and automatically redirects the Writer Endpoint to it — the application needs no configuration changes.

Q2: What are the three ElastiCache caching patterns and when would you use each?
Lazy Loading caches data only on a cache miss — good for read-heavy workloads where some staleness is acceptable. Write Through updates both the cache and database on every write — good when consistency is critical like banking or inventory. Session Store uses the cache with a TTL to hold temporary session data like login sessions and shopping carts — not persisted to a database at all.

Q3: How do you handle stale data in a Lazy Loading cache?
Explicitly delete the cache entry whenever the underlying database record is updated. The next read triggers a cache miss, fetches the latest data from the database, and refreshes the cache. A TTL on cache entries provides automatic backup expiry even if explicit invalidation is missed.

Q4: What is the difference between a Query and a Scan in DynamoDB?
A Query uses the Partition Key to efficiently retrieve only the relevant items — fast, cheap, and the right choice for production. A Scan reads the entire table across all partitions before filtering — slow, expensive, and charged for all data read regardless of what the filter returns. In production, Scans should almost never be used.

Q5: What is the difference between an LSI and a GSI?
An LSI uses the same Partition Key as the base table but a different Sort Key — it allows alternative sorting within a single partition and must be created at table creation time. A GSI has its own completely independent Partition Key (and optional Sort Key), can query across all partitions using a different attribute, and can be created at any time after the table exists. GSIs have separate throughput (additional cost); LSIs share the base table's throughput.

Q6: Can you add a Local Secondary Index after a DynamoDB table is created?
No. LSIs must be defined at table creation time. If you need one after the fact, you'd have to create a new table with the LSI defined and migrate the data. GSIs can be added at any time, making them far more flexible when requirements change.

Q7: When would you choose On-Demand capacity mode over Provisioned for DynamoDB?
On-Demand is best for unpredictable or spiky traffic, new products where read/write patterns aren't yet known, and development/testing environments. Provisioned capacity is better for steady, predictable production workloads where you know the baseline RCU/WCU requirements — it's cheaper per unit when fully utilized, but throttles if traffic exceeds the provisioned amount.

Q8: What is the difference between DAX and ElastiCache for DynamoDB caching?
DAX is a DynamoDB-native in-memory cache that uses the same API as DynamoDB and requires no application code changes, reducing latency from milliseconds to microseconds. ElastiCache is a general-purpose caching layer (Redis/Valkey/Memcached) that can cache data from DynamoDB or any other source, but requires application-level code to check the cache and handle misses. DAX is simpler for pure DynamoDB acceleration; ElastiCache offers more flexibility for multi-source caching.

🔬 Practice Tasks

Aurora Endpoints lab: Create an Aurora MySQL cluster with 2 replicas. Connect to the Writer Endpoint and insert some rows. Connect to the Reader Endpoint and SELECT the same rows — confirm it serves reads. Manually trigger a failover and verify the Writer Endpoint automatically redirects to the new primary within ~30 seconds.
Caching pattern implementation: Using Python and boto3, implement Lazy Loading against a DynamoDB table with an ElastiCache Redis cluster. Log every cache hit and miss. Observe the cache hit rate improve as the same keys are read repeatedly.
DynamoDB design exercise: You're building a messaging app. Users send messages to other users. Design the DynamoDB table: what's your partition key? Sort key? What LSI or GSI would you add to support "show all messages I received, sorted by timestamp"?
Query vs Scan cost experiment: Create a DynamoDB table with 1000 items. Run a Query for a specific partition key — note the consumed RCUs. Run a Scan with a filter returning the same single item — note the consumed RCUs. Compare. The Scan should show RCU consumption for all 1000 items, not just the one that matched.
GSI creation on existing table: Create a DynamoDB table with partition key = user_id. Add items. After creation, add a GSI with partition key = city. Query by city and confirm it works without touching the base table's key structure.

AWS Session 12 — Aurora, ElastiCache Patterns & DynamoDB | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS RDS & ElastiCache — Managed Databases, Multi-AZ Failover & In-Memory Caching

Tejas Shinkar — Wed, 05 Aug 2026 13:06:28 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session moves from networking into the data layer — how AWS manages databases for you, and how caching keeps applications fast.

📋 Topics Covered

#	Topic	Type
1	Why Separate Compute from Database	Concept
2	Structured vs Semi-Structured vs Unstructured Data	Concept + Interview
3	What RDS Actually Is Under the Hood	Concept + Interview
4	RDS Deployment Options — Single-AZ vs Multi-AZ Instance vs Multi-AZ Cluster	Concept + Cert
5	Self-Managed DB vs RDS — What AWS Automates	Concept + Interview
6	Multi-AZ Failover — Step by Step	Concept + Cert
7	Why RDS Uses a DNS Endpoint, Not an IP	Interview
8	Amazon Aurora	Concept + Cert
9	Sharding — Horizontal Scaling	Concept + Interview
10	Amazon ElastiCache	Concept + Interview
11	Lab — Creating an RDS Database	Lab
12	Interview Questions	Interview
13	Assignment	Practice

Why Separate Compute from Database

An application server (EC2, Lambda) is built to process data quickly — run logic, transform data, respond to requests. A database is built to store data reliably — persist it, keep it searchable, keep it available to multiple clients at once, and survive restarts.

If you put both on the same machine, a crash takes down your entire system — your data and your processing logic disappear together. Keeping them separate means your application servers can scale up, scale down, or crash and restart, while your data stays safe and available on its own dedicated, purpose-built infrastructure.

The core principle: Compute is optimized for processing. A database is optimized for persistent, reliable, searchable, shared storage. Separating them lets each scale independently and lets your application survive server failures without losing data.

Structured vs Semi-Structured vs Unstructured Data

Not all data looks the same, and AWS has a different storage service optimized for each type.

Data Type	Schema	Example	AWS Service
Structured	Fixed schema — rows and columns defined upfront	Customer table: id, name, email, order_date	Amazon RDS (relational)
Semi-structured	Flexible schema — fields can vary per record	JSON, XML — a user profile with optional fields	Amazon DynamoDB (NoSQL)
Unstructured	No predefined format at all	Images, videos, PDFs, log files	Amazon S3

Why this distinction matters in real architecture: A retail app might use RDS for orders (structured, needs strong consistency), DynamoDB for a shopping cart (semi-structured, needs speed and flexibility), and S3 for product images (unstructured, needs cheap durable storage). Picking the right store for each data type is a core system design skill.

What RDS Actually Is Under the Hood

This mental model makes RDS click immediately: RDS is not a magical new thing — it's an EC2 instance with attached EBS storage, running a database engine, wrapped in an AWS-managed control plane.

Think of it this way: underneath, RDS is still a virtual machine with a disk, just like any EC2 setup you could build yourself. What AWS adds on top is the automation layer — automatic provisioning, backups, patching, monitoring, scaling, and failover — so you never have to SSH in and manage the database server yourself.

What you get with RDS that you'd have to build yourself with self-managed EC2:

Task	Self-Managed on EC2	Amazon RDS
Install DB engine	Manual	Pre-configured, just choose the engine
OS/DB patching	Manual, scheduled downtime	Automated maintenance windows
Backups	You script and schedule them	Automated daily snapshots + point-in-time recovery
Failover	You build and test it yourself	Automatic (Multi-AZ)
Monitoring	You set up CloudWatch agents	Built-in metrics out of the box
Scaling storage	Manual resize + downtime	Can auto-scale storage

RDS Deployment Options

RDS offers three deployment models, and this is one of the most tested concepts in the SAA-C03 exam. The image from class (RDS Console's deployment screen) shows this exact choice.

Memory trick:

Single-AZ → One copy 🏠
Multi-AZ Instance → One primary + one standby 🏠🏠
Multi-AZ DB Cluster → One writer + multiple readers 🏠🏠🏠

Side-by-Side Comparison

	Single-AZ	Multi-AZ DB Instance	Multi-AZ DB Cluster
Instances	1	2 (1 primary + 1 standby)	3 (1 writer + 2 readable standbys)
Uptime SLA	99.5%	99.95%	99.95%
Automatic failover	❌ No	✅ Yes	✅ Yes
Standby readable	N/A	❌ No (standby is idle, failover-only)	✅ Yes (readers serve read traffic)
Read scaling	❌ No	❌ No	✅ Yes
Redundancy across AZs	❌ No	✅ Yes	✅ Yes
Use case	Dev, test, non-critical workloads	Production, HA required	Production, HA + read-heavy workloads
Cost	Lowest	Medium (2x compute)	Highest (3x compute)

Reading the deployment screenshot from class:

Multi-AZ DB Cluster (3 instances): Primary instance + SSD in AZ 1, two "Readable standby + SSD" instances in AZ 2 and AZ 3. There's a separate "Reader endpoint" that load-balances read queries across the standbys, while the "Write/read endpoint" always points to the primary.
Multi-AZ DB Instance (2 instances): Primary in AZ 1, a standby in AZ 2 that has no endpoint of its own — it exists purely as a failover target, not for serving traffic.
Single-AZ (1 instance): Just the primary, no redundancy at all. If AZ 1 has an issue, there's no automatic failover — you'd be restoring from backup.

🎯 Cert tip: The key differentiator that trips people up — in a Multi-AZ Instance deployment, the standby is NOT readable. You cannot send read queries to it; it exists purely for failover. Only a Multi-AZ DB Cluster gives you readable standbys that also help with read scaling.

Self-Managed DB vs RDS — What AWS Automates

On a self-managed database (say, PostgreSQL installed manually on an EC2 instance), a DBA is responsible for installing the database binaries, configuring database parameters (memory allocation, connection limits, query optimization settings), and managing how clients connect.

RDS automates all three of these:

Self-Managed Task	RDS Equivalent
Manually install DB binaries	Choose an engine (PostgreSQL, MySQL, etc.) — AWS manages the binary
Manually edit config files (`postgresql.conf`)	Parameter Groups — a managed collection of engine settings you can tune without SSH access
Manage static connection strings, update on every failover	DNS Endpoint — a stable hostname that AWS keeps pointed at the current primary

Parameter Groups deserve a specific callout — instead of editing a config file directly on the server, you modify settings through a Parameter Group in the RDS Console, and AWS applies them to the instance (sometimes requiring a reboot, depending on the parameter).

Multi-AZ Failover — Step by Step

This is the mechanism that makes Multi-AZ deployments valuable — understanding exactly what happens during a failure builds real confidence for both interviews and production incidents.

The failover sequence:
Primary Crash → Standby Promoted → DNS Endpoint Updated (TTL = 5 seconds) → Application Resolves New IP → Reconnect → Database Available

Total time: roughly 60–120 seconds

Breaking down each step:

Primary Crash — the primary instance becomes unreachable (hardware failure, AZ outage, or a manual failover for maintenance)
Standby Promoted — AWS automatically promotes the standby replica in the other AZ to become the new primary
DNS Endpoint Updated — the RDS DNS endpoint's record is updated to point to the new primary's IP address, with a TTL (Time To Live) of about 5 seconds
Application Resolves New IP — because the DNS TTL is so short, client applications quickly pick up the new IP on their next DNS lookup
Reconnect — the application's connection pool reconnects using the newly resolved IP
Database Available — normal operations resume

The entire process typically completes within 60 to 120 seconds — no manual intervention required.

Why RDS Uses a DNS Endpoint, Not an IP

This is a genuinely good interview question because it tests whether you understand why a design choice was made, not just what it is.

Q: Why does RDS use a DNS endpoint instead of exposing the database IP directly?

During a Multi-AZ failover, the database's IP address changes — the standby gets promoted and it has a different IP than the old primary. If applications connected using a hardcoded IP, every failover would require someone to manually update every application's configuration and restart it — completely defeating the purpose of "automatic" failover.

Instead, applications connect using a stable RDS DNS endpoint (something like mydb.abc123xyz.ap-south-1.rds.amazonaws.com) that never changes. Behind the scenes, AWS updates what that DNS name resolves to. With a TTL of about 5 seconds, client applications re-resolve the DNS quickly and reconnect to the new primary automatically — with zero configuration changes needed on the application side.

This is exactly the same principle used by Elastic IPs and Route 53 health checks elsewhere in AWS — decouple the stable identifier from the underlying resource so the underlying resource can change without breaking anything upstream.

Amazon Aurora

Amazon Aurora is AWS's own cloud-native relational database, compatible with MySQL and PostgreSQL (meaning your existing MySQL/PostgreSQL drivers and tools work without changes) but built with fundamentally different internal architecture for much higher performance.

What makes Aurora different from standard RDS engines:

Feature	Standard RDS (MySQL/PostgreSQL)	Aurora
Compute + Storage	Coupled together	Separated — storage scales independently
Replication	You configure it	Automatic, across 3 AZs, 6 copies of data
Storage auto-scaling	Manual/limited	Automatic, up to 128 TB
Failover speed	~60-120 seconds	Faster (typically under 30 seconds)
Performance	Baseline	Significantly higher throughput
Cost	Lower	Higher (premium for the performance)

The "separates compute from storage" concept, explained simply: In a standard database, if you need more storage, you often need to resize the whole instance. In Aurora, the storage layer is a separate, distributed system that grows automatically as your data grows — the compute instance (which runs the actual query engine) can scale independently. This is why Aurora can offer both faster failover (the storage layer already has 6 copies ready) and larger scale (storage isn't tied to a single disk).

🎯 Cert tip: When a scenario mentions "MySQL-compatible," "PostgreSQL-compatible," "high performance," and "automatic storage scaling" together, the answer is almost always Aurora, not standard RDS.

Sharding — Horizontal Scaling for Databases

As a database grows, at some point one server (even a very large one) can't handle the write load or storage anymore. Sharding is the technique for scaling out — instead of one giant database, you split it into multiple smaller databases called shards, each holding a portion of the data.

How it works: A large database is partitioned into multiple smaller databases (shards) based on some key — for example, splitting users A-M into Shard 1 and users N-Z into Shard 2. Each shard stores only its subset of the data. The application (or a routing layer) determines which shard to query based on the data being requested, so both storage and write load get distributed across multiple database servers instead of one.

Why this matters: Vertical scaling (bigger instance) has a ceiling — eventually you run out of bigger instance types. Sharding is how systems scale writes and storage beyond what any single database server could handle, at the cost of added application complexity (your app needs to know which shard to query).

💡 Where this connects: This is conceptually the same "horizontal vs vertical scaling" idea from the ELB/ASG session — just applied to databases instead of compute. Vertical = bigger box. Horizontal (sharding) = more boxes, each handling a slice of the problem.

Amazon ElastiCache

Even a well-tuned database has a limit to how many reads it can serve per second, and every query — even a fast one — has some latency. ElastiCache is AWS's fully managed in-memory caching service — it stores frequently accessed data in RAM, which is dramatically faster than querying a database on disk.

How caching works in an application:

Application needs data → checks the cache first → if the data is there (cache hit), return it instantly from RAM → if the data isn't there (cache miss), query the database, get the result, store it in the cache for next time, then return it to the caller.

This means the first request for a piece of data is a normal database query, but every subsequent request for the same data is served from memory — orders of magnitude faster, and it takes load off the database entirely.

Three Caching Engines

Engine	Persistence	Replication	Status
Valkey	✅ Yes	✅ Yes	Open-source fork of Redis (community-driven, actively used going forward)
Redis OSS	✅ Yes	✅ Yes	Long-standing standard, still widely used
Memcached	❌ No	❌ No	Simpler, but largely fallen out of favor — no persistence or replication means data loss on restart

💡 Why Valkey exists: After a licensing change to Redis, the open-source community forked the last fully open-source version of Redis into a new project called Valkey — it's Linux Foundation-backed and functionally very similar to Redis. AWS supports it as a first-class ElastiCache engine going forward.

Memcached is essentially legacy at this point — no persistence means a restart wipes your cache entirely, and no replication means no high availability. Almost all new projects choose Redis OSS or Valkey.

🧪 Lab — Creating an RDS Database (Progress So Far)

This is the practical work completed so far in class — continuing in a future session to connect it to EC2 and build the full application.

Step 1 — Create a DB Subnet Group

A DB Subnet Group tells RDS which subnets (across which AZs) it's allowed to place database instances in.

Go to RDS Console → Subnet Groups → Create DB Subnet Group. Choose your existing VPC. Select the private subnets across each of the 3 Availability Zones — databases should never sit in a public subnet.

Step 2 — Create the Database

RDS Console → Create database → chose the PostgreSQL engine → selected the engine version → set a DB instance identifier (a name for this specific database instance) → set the master username → chose self-managed password authentication → selected the DB instance class (Burstable class, e.g., db.t3.micro) → chose storage type and allocated storage size → left storage auto-scaling disabled for now.

Step 3 — Connectivity

Chose "Connect to an EC2 compute resource" during setup → this triggered creating an EC2 instance to connect to the database → created an Internet Gateway and attached it to the VPC → configured subnet routes → configured connectivity, including an additional VPC security group (launch-wizard-2) → disabled Performance Insights for now (cost-saving on a lab account) → DB Subnet Group was auto-selected based on the earlier subnet group creation → clicked Create Database.

What's left for the next session: actually connecting from the EC2 instance to the RDS database, verifying connectivity, and building out the application layer on top.

⚡ Quick Revision

Why Separate Compute & DB
Compute = optimized for processing. Database = optimized for persistent, reliable, shared storage. Separation lets each scale independently and survive failures.

Data Types
Structured (fixed schema) → RDS. Semi-structured (flexible schema, JSON/XML) → DynamoDB. Unstructured (no format) → S3.

What RDS Really Is
EC2 + EBS + DB engine + AWS-managed automation layer (provisioning, backups, patching, monitoring, scaling, failover).

Three Deployment Options

Single-AZ 🏠 → one instance, no failover, 99.5% SLA
Multi-AZ Instance 🏠🏠 → primary + non-readable standby, automatic failover, 99.95% SLA
Multi-AZ Cluster 🏠🏠🏠 → one writer + 2 readable standbys, failover + read scaling, 99.95% SLA

Failover Flow
Primary Crash → Standby Promoted → DNS Updated (TTL 5s) → App Resolves New IP → Reconnect → Available. Total: 60-120 seconds.

Why DNS Endpoint, Not IP
IP changes on failover. DNS endpoint stays constant; AWS updates what it resolves to. Apps never need reconfiguration.

Aurora
MySQL/PostgreSQL-compatible, but compute and storage are separated. 3 AZs, 6 copies of data automatically. Auto-scales storage to 128 TB. Faster failover, higher performance than standard RDS.

Sharding
Horizontal scaling for databases — split one large DB into multiple shards, each holding a subset of data, distributing storage and write load.

ElastiCache
In-memory caching. Cache hit = instant from RAM. Cache miss = query DB, store result in cache, return. Engines: Valkey (Redis fork, active), Redis OSS (standard), Memcached (legacy, no persistence/replication).

💼 Interview Questions

Q1: Why does AWS keep compute and database storage separate instead of running everything on one server?
Compute is optimized for processing logic quickly, while databases are optimized for persistent, reliable, and shared storage. Keeping them separate allows application servers to scale independently, restart, or fail without losing data, since the database lives on its own dedicated, durable infrastructure.

Q2: What is the difference between a Multi-AZ DB Instance and a Multi-AZ DB Cluster?
A Multi-AZ DB Instance has one primary and one standby — the standby is not readable and exists purely for automatic failover. A Multi-AZ DB Cluster has one writer and two readable standbys across different AZs, providing both automatic failover and read scaling, since the standbys can serve read traffic through a separate reader endpoint.

Q3: Walk through what happens during an RDS Multi-AZ failover.
The primary instance becomes unavailable, so AWS automatically promotes the standby in another AZ to primary. The RDS DNS endpoint's record is updated to point to the new primary's IP, with a TTL of about 5 seconds. Applications re-resolve the DNS quickly and reconnect using the new IP — no manual configuration changes needed. The whole process typically takes 60 to 120 seconds.

Q4: Why does RDS use a DNS endpoint instead of a static IP address?
Because the underlying IP address changes during failover — the standby that gets promoted has a different IP than the old primary. If applications connected via a hardcoded IP, every failover would require manual reconfiguration. The DNS endpoint stays constant while AWS updates what it resolves to, with a short TTL so clients pick up the change within seconds automatically.

Q5: What makes Amazon Aurora different from standard RDS engines like MySQL or PostgreSQL on RDS?
Aurora separates compute from storage — the storage layer is a distributed system that automatically replicates across 3 AZs with 6 copies of data and scales up to 128 TB automatically. This architecture also enables faster failover and significantly higher performance compared to standard RDS engines, while remaining compatible with existing MySQL and PostgreSQL tooling.

Q6: What is sharding and when would you use it?
Sharding is a horizontal scaling technique where a large database is split into multiple smaller databases (shards), each storing a subset of the data. It's used when a single database instance can no longer handle the write load or storage requirements — even with vertical scaling — because it distributes both storage and write traffic across multiple servers.

Q7: How does ElastiCache improve application performance?
Applications check the cache before querying the database. On a cache hit, data is returned instantly from RAM, which is dramatically faster than a database query. On a cache miss, the application queries the database, then stores the result in the cache for future requests. This reduces database load and significantly improves response times for frequently accessed data.

Q8: Why is Memcached rarely chosen for new projects compared to Redis OSS or Valkey?
Memcached has no persistence (data is lost on restart) and no replication (no high availability). Redis OSS and Valkey both support persistence and replication, making them suitable for production caching layers where data durability and availability matter — which is why most new projects choose one of those two instead.

🔬 Assignment

Create a web server on EC2, host a simple application on it, connect it to the RDS database created in this session's lab, build a basic frontend, and create a form that makes an entry into the database.

AWS Session 11 — RDS & ElastiCache | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer
Practical RDS-to-EC2 connection and remaining lab steps to be completed in a follow-up practical session.

AWS VPC Extended — Route Tables, Blackhole Routes, Flow Logs, VPC Endpoints & Transit Gateway

Tejas Shinkar — Fri, 31 Jul 2026 09:11:31 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This builds directly on the previous VPC session — going deeper into how routing actually decides where traffic goes, and how VPCs connect privately to AWS services and to each other.

This is dense, interconnected material — the goal here isn't to memorize every term on the first pass, but to build a reference to come back to once the full picture settles.

📋 Topics Covered

#	Topic	Type
1	Route Table — The Traffic Director	Concept
2	Destination vs Target	Concept + Analogy
3	Longest Prefix Match — Deep Dive with Scenarios	Concept + Interview
4	Route Propagation	Concept + Cert
5	Static Routes vs Propagated Routes	Concept + Interview
6	Blackhole Routes	Concept + Interview
7	Managed Prefix Lists	Concept + DevOps
8	Edge Ingress Route Table Association	Concept + Cert
9	Troubleshooting Checklist — Traffic Not Reaching EC2	Interview + DevOps
10	VPC Flow Logs	Concept + Lab
11	VPC Endpoints — Gateway vs Interface	Concept + Interview
12	VPC Endpoint Policy	Concept + Cert
13	Transit Gateway	Concept + DevOps
14	Interview Questions	Interview
15	Practice Scenarios	Practice

Route Table — The Traffic Director

A Route Table is the rulebook that decides where every packet goes next, based purely on its destination IP address. Every subnet is associated with exactly one route table, and every route table contains one or more rules.

Analogy: A route table is GPS navigation for your network. You type in a destination (IP address), and the GPS tells you which road (target) to take. If multiple roads could technically get you there, GPS picks the most direct, specific route — not the broadest general highway.

Basic structure:

Destination	Target
10.0.0.0/16	Local
0.0.0.0/0	nat-0abc123

Destination vs Target — The Core Distinction

This is the single most important mental model for understanding routing.

Destination = WHERE you're going (the IP range you're trying to reach)
Target = HOW you get there (which gateway/component forwards it)

The analogy:

Destination: Mumbai → Target: Mumbai-Pune Expressway
Destination: Pune → Target: Local roads (already there)
Destination: Anywhere else → Target: National Highway (catch-all)

In AWS terms:

Destination	Target	Meaning
10.200.0.0/16	Local	VPC's own network — no gateway needed
0.0.0.0/0	igw-0a1b2c3d	Anywhere else — go through Internet Gateway
172.31.0.0/16	pcx-0x1y2z3	Peered VPC — go through Peering Connection
192.168.0.0/16	vgw-0p1q2r3	On-premise network — go through VPN Gateway

Possible targets in a route table:

Target	What it connects to
`local`	Always present — traffic within the VPC's own CIDR
Internet Gateway (`igw-`)	Public internet
NAT Gateway (`nat-`)	Internet, for private subnet outbound only
VPC Peering Connection (`pcx-`)	Another VPC (peered)
Virtual Private Gateway (`vgw-`)	On-premise network via VPN
Transit Gateway (`tgw-`)	Multiple VPCs/networks via central hub
VPC Endpoint	Specific AWS service, privately

Longest Prefix Match — Deep Dive with Scenarios

This is the rule AWS uses whenever a destination IP could match more than one route in the table. AWS always picks the route with the most specific (longest) prefix — not the broadest one.

Core rule: A more specific CIDR (larger number after the /) always wins over a broader CIDR (smaller number). /32 beats /24 beats /16 beats /0 (most specific → least specific catch-all).

Scenario Comparisons:

Scenario 1 — Basic internet vs local

Route Table	Traffic to 10.200.0.55	Traffic to 8.8.8.8
`10.200.0.0/16 → Local` `0.0.0.0/0 → IGW`	Matches `/16` (more specific) → Local ✅	Doesn't match `/16` Matches `/0` (catch-all) → IGW ✅

Because 8.8.8.8 doesn't fall within the VPC's CIDR, the only matching route is the default 0.0.0.0/0, so it goes to the Internet Gateway.

Scenario 2 — Peered VPC with overlapping catch-all

Route Table	Traffic to 172.31.5.10	Traffic to 142.250.1.1 (Google)
`10.200.0.0/16 → Local` `172.31.0.0/16 → pcx-xyz` `0.0.0.0/0 → IGW`	Matches `/16` peering route → Peering Connection ✅	No specific match Falls to `/0` catch-all → IGW ✅

The peering route is the only specific match for the peered VPC's range — it wins over the general internet catch-all route.

Scenario 3 — Specific /24 override within a /16

Route Table	Traffic to 10.200.5.20	Traffic to 10.200.1.5
`10.200.0.0/16 → Local` `10.200.5.0/24 → tgw-abc` `0.0.0.0/0 → IGW`	Matches BOTH `/16` and `/24` `/24` is more specific → Transit Gateway ✅	Only matches `/16` → Local ✅

This is the key exam trap: even though 10.200.5.20 falls inside the broader /16 local range, the more specific /24 route to the Transit Gateway wins. AWS always prefers the longest (most specific) prefix, regardless of which route was added first.

Scenario 4 — VPN + Internet coexisting

Route Table	Traffic to 192.168.10.5 (on-prem)	Traffic to Facebook.com
`10.200.0.0/16 → Local` `192.168.0.0/16 → vgw-onprem` `0.0.0.0/0 → IGW`	Matches `/16` VPN route → VPN Gateway ✅	No specific match → IGW ✅

🎯 Interview trap to remember: Longest Prefix Match is about specificity of the CIDR, not the order routes were added, not alphabetical order, not which target "seems more important." Just count the prefix length — bigger number after the / always wins.

Route Propagation

Static routes are ones you manually type into a route table. Propagated routes are learned automatically — AWS discovers them from a dynamic routing source and adds them to your route table without manual entry.

How it flows:

On-Premise Network → (VPN or Direct Connect) → Virtual Private Gateway / Transit Gateway → (BGP advertises routes) → AWS learns the on-prem CIDR ranges automatically → Route Table (with Route Propagation enabled) → Routes to on-premise networks appear automatically, no manual entry, updates as the on-prem network changes.

Why this matters: Without propagation, every time your on-premise network team adds a new subnet, someone would have to manually update every AWS route table. With propagation enabled, AWS learns the change automatically via BGP.

Static Routes vs Propagated Routes — Conflict Resolution

What happens if a static route and a propagated route both match the same destination with the same prefix length?

Rule: Longest Prefix Match is applied first. If multiple routes have the same prefix length, the static route wins over the propagated route.

Example:

Route Type	Destination	Target
Static (you typed this)	10.50.0.0/16	pcx-manual
Propagated (learned via BGP)	10.50.0.0/16	vgw-learned

Same prefix length (/16) → the static route wins. Traffic to 10.50.5.5 goes through pcx-manual, not vgw-learned.

This gives you an override mechanism — if AWS learns a route via propagation that conflicts with what you want, you can add a static route with the same prefix to take priority.

Blackhole Routes

A Blackhole Route is a route that still exists in the table, but its target no longer exists or is unavailable. Traffic matching this route gets silently dropped — packets vanish into nowhere, hence "blackhole."

Most common cause:

Route table has 0.0.0.0/0 → nat-0abc123. Someone deletes the NAT Gateway (nat-0abc123). The route table still has 0.0.0.0/0 → nat-0abc123 — now shown as Blackhole. Any traffic matching 0.0.0.0/0 is dropped silently, with no error sent back to the sender.

Analogy: It's like your GPS still showing directions to a building that was demolished last week. The road exists, the GPS still routes you there, but there's nothing at the destination — you just vanish into a void.

How to spot it: In the AWS Console, Route Tables show a "Blackhole" status directly next to the affected route — AWS flags it visually so you don't have to guess.

Common triggers:

NAT Gateway deleted while route table still references it
VPC Peering connection deleted/rejected while routes still point to it
VPN connection terminated while VGW routes remain
Transit Gateway attachment removed

🎯 Classic interview question: "Your EC2 in a private subnet suddenly cannot access the internet, but it was working yesterday. What would you check?"

Answer — the troubleshooting sequence:

Is the NAT Gateway still available? (check EC2 → NAT Gateways console)
Does the route table still point to the correct NAT Gateway ID?
Is the route marked "Blackhole"? (visible directly in Console)
Are Security Groups and NACLs still allowing the traffic?

Managed Prefix Lists

Imagine you have 20 Security Groups and 10 Route Tables, all referencing the same set of 5 office IP ranges. If those office IPs change, you'd have to update 30 different places manually.

A Managed Prefix List solves this — it's a named, reusable group of CIDR blocks you can reference in Security Groups and Route Tables. Update the list once, and every place referencing it gets the update automatically.

Example — Prefix List "company-offices":

Office	CIDR
Mumbai	10.1.0.0/24
Bangalore	10.2.0.0/24
Pune	10.3.0.0/24

You'd write a Security Group rule as "Allow SSH from pl-company-offices" instead of listing all three CIDRs individually. Add a new office IP → update the Prefix List once → every Security Group and Route Table referencing it updates automatically. ✅

AWS also provides pre-built Managed Prefix Lists for its own services — e.g., a prefix list containing all CloudFront IP ranges, so you don't have to track them manually as AWS's infrastructure changes.

💡 DevOps relevance: Managed Prefix Lists are a genuine time-saver in infrastructure-as-code — instead of hardcoding IP ranges across dozens of Terraform resources, reference one prefix list ARN.

Edge Ingress Route Table Association

This is an advanced feature — most engineers won't touch it often, but it's worth understanding conceptually.

Normally, route tables are associated with subnets. Edge Ingress lets you associate a route table with an Internet Gateway or Virtual Private Gateway itself — meaning you can intercept and redirect traffic the moment it enters your VPC, before it even reaches a subnet.

Without Edge Ingress: Internet → IGW → directly to subnet → EC2

With Edge Ingress: Internet → IGW → Edge Route Table → redirected through a Firewall/Inspection Appliance → then to subnet → EC2

Use case: Route all inbound internet traffic through a third-party firewall or intrusion detection appliance before it reaches your actual application servers — a centralized security inspection point at the very edge of your VPC.

🎯 Cert-level awareness only: You're unlikely to configure this often in early-career roles, but knowing it exists (and what problem it solves — centralized traffic inspection at the VPC edge) is enough for interviews and certification.

Troubleshooting Checklist — Traffic Not Reaching EC2

This is the master checklist that ties everything from both VPC sessions together. When traffic isn't reaching an EC2 instance, work through this in order:

Route Table Association — Is the correct route table associated with this subnet? (or with the IGW, in edge ingress scenarios)
Longest Prefix Match — Is AWS selecting the route you expect it to select? Check for a more specific route accidentally overriding your intended path.
Target Status — Is the target still valid? Is the route marked "Blackhole"? (target deleted/unavailable)
Address Translation Path — Via Internet Gateway: does the EC2 have a Public IP or Elastic IP? Via NAT Gateway: is the NAT Gateway available, and is the route pointing to it correctly?
Security Layer — Are Security Groups allowing the traffic? (stateful) Are NACLs allowing the traffic? (stateless — check BOTH inbound AND outbound, including ephemeral ports 1024-65535 for return traffic)

💡 This exact sequence — Route Table → Prefix Match → Target Status → Address Translation → Security — is the professional troubleshooting flow used in real production incidents. Memorizing this order (not just the individual pieces) is what makes you effective in an actual outage, not just in an exam.

VPC Flow Logs

VPC Flow Logs capture metadata about IP traffic flowing to and from network interfaces in your VPC. This is your forensic tool — when something goes wrong (or when security needs to investigate), Flow Logs tell you exactly what traffic happened, where, and whether it was allowed or blocked.

What Flow Logs capture (metadata, NOT packet content):

Field	Meaning
Source IP	Who sent it
Destination IP	Who it was sent to
Source Port	Which port on sender
Destination Port	Which port on receiver
Protocol	TCP, UDP, ICMP
Packets & Bytes	Volume of traffic
Action	ACCEPT or REJECT
Timestamp	When it happened

💡 Flow Logs capture metadata only — not the actual content/payload of the traffic. You'll know that 10.200.1.5 sent 500 bytes to 8.8.8.8 on port 443 and it was accepted — but not what was inside that HTTPS request.

Where Flow Logs can be published: VPC / Subnet / ENI → sends a Flow Log to either CloudWatch Logs (real-time monitoring, alarms, dashboards) or an S3 Bucket (long-term storage, cost-effective, Athena queries).

Setup sequence (what was done in class):

Create a Log Group in CloudWatch → CloudWatch → Log groups → Create log group → name it /vpc/flowlogs
Create the Flow Log → VPC Console → Your VPC → Flow Logs → Create flow log → set filter (All / Accept / Reject) → destination: CloudWatch Logs → select the log group → attach an IAM role with permission to write logs
Watch logs flow in → CloudWatch → Log groups → /vpc/flowlogs → Log streams → real traffic metadata appears

Sample Flow Log entry (this one is real syntax, kept as code):

2 123456789010 eni-1235b8ca 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK

Breaking this down: version, account ID, ENI ID, source IP, dest IP, source port, dest port, protocol (6=TCP), packets, bytes, start time, end time, action (ACCEPT), log status.

🎯 Security use case: If you suspect unauthorized access, filter Flow Logs for REJECT actions from unfamiliar IPs — this shows every blocked connection attempt, which is exactly what a security investigation needs. This is one of the most useful forensic tools in AWS networking.

VPC Endpoints — Private Connectivity to AWS Services

Normally, when a private EC2 instance wants to talk to S3, DynamoDB, or another AWS service, that traffic would need to go through a NAT Gateway → Internet Gateway → out to the public internet → to the AWS service's public endpoint → back.

A VPC Endpoint creates a private connection directly from your VPC to the AWS service — traffic never leaves the AWS network, never touches the public internet, no NAT Gateway or Internet Gateway required.

The analogy:

🏠 Private EC2 = Your house
🏢 AWS Service (S3, SSM, CloudWatch) = Office

Without a VPC Endpoint: House → Public Road (internet, via NAT + IGW) → Office. Longer path, exposed to the public internet, costs a NAT data processing fee.

With a VPC Endpoint: House → Private Tunnel (stays inside AWS network) → Office. Shorter, more secure, often cheaper — no NAT processing charge.

Two Types of VPC Endpoints

Gateway Endpoint

Only supports S3 and DynamoDB
Works by adding a route table entry — a target pointing to the endpoint
Free — no hourly charge
Cannot be accessed from on-premises networks (VPN/Direct Connect)

Route table example: destination pl-xxxxxx (the S3 prefix list) → target vpce-gateway-xxxxx. A private EC2 making an S3 request matches the S3 prefix list route, goes through the Gateway Endpoint, and reaches S3 — no internet involved.

Interface Endpoint

Supports most other AWS services (SSM, CloudWatch, SNS, SQS, KMS, Secrets Manager, etc.)
Works by creating an ENI with a private IP inside your subnet
Has an hourly charge + data processing charge
Can be accessed from on-premises networks via VPN/Direct Connect
Optionally enables Private DNS — makes the standard AWS service DNS name resolve to the private endpoint automatically

Example: an Interface Endpoint for SSM creates an ENI at 10.200.1.100 in your private subnet. With Private DNS enabled, ssm.ap-south-1.amazonaws.com resolves to 10.200.1.100 instead of the public AWS IP. So when your EC2 calls the standard SSM endpoint name, DNS quietly resolves it privately, and traffic stays inside the VPC.

Gateway vs Interface — Comparison

	Gateway Endpoint	Interface Endpoint
Supported services	S3, DynamoDB only	Most other AWS services
How it works	Route table entry	ENI with private IP
Cost	Free	Hourly + data charge
On-premise access	❌ No	✅ Yes (via VPN/DX)
DNS	N/A	Private DNS optional

The Interview Trap — Gateway vs Interface for S3

🎯 Interviewer: "You have both an S3 Gateway Endpoint AND an S3 Interface Endpoint, and Private DNS is enabled on the Interface Endpoint. Which one actually gets used?"

Answer: The Interface Endpoint is used. Enabling Private DNS overrides the default S3 DNS resolution — so when your application calls the standard S3 endpoint name (s3.ap-south-1.amazonaws.com), DNS resolves it to the Interface Endpoint's private IP instead of routing through the Gateway Endpoint via the route table. The Gateway Endpoint sits unused in this scenario even though it's still technically active.

This is a genuinely tricky, DNS-resolution-based gotcha — worth remembering exactly because it's counterintuitive.

VPC Endpoint Policy

Just like S3 has bucket policies and IAM has permission policies, a VPC Endpoint Policy is a resource policy attached directly to the endpoint itself — controlling exactly which AWS resources and API actions can flow through that specific endpoint.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": ["s3:GetObject"],
    "Resource": "arn:aws:s3:::company-approved-bucket/*"
  }]
}

This policy says: "Through this specific VPC Endpoint, only GetObject on this one specific bucket is allowed — nothing else, even if IAM would otherwise permit it."

Why this matters: It's an additional restriction layer, not a replacement. A request must pass IAM policy AND the VPC Endpoint policy AND (if applicable) the bucket policy. This lets you say: "Even if someone has full S3 permissions via IAM, when they go through THIS endpoint, they can only touch THIS bucket."

🎯 Production use case: A company wants private subnets to reach only their own company's S3 buckets via the Gateway Endpoint — not any random public S3 bucket on the internet. The VPC Endpoint Policy restricts this precisely.

Transit Gateway

As companies grow, they end up with many VPCs — one per team, per environment (dev/staging/prod), per project. Connecting all of them directly to each other via VPC Peering becomes a mess very quickly.

The peering problem:

VPC Count	Peering Connections Required
5 VPCs	10 connections
10 VPCs	45 connections
20 VPCs	190 connections

Each peering connection needs its own route table entries everywhere. This does not scale.

Transit Gateway solves this — it's a fully managed, regional network hub that all your VPCs connect to once, and it handles routing between them.

Without Transit Gateway (Peering Mesh): VPC-A ↔ VPC-B ↔ VPC-C ↔ VPC-D — every VPC needs a direct connection to every other VPC.

With Transit Gateway (Hub and Spoke): VPC-A, VPC-B, VPC-C, and VPC-D each connect once to the central Transit Gateway. The Transit Gateway routes traffic between all of them centrally.

What Transit Gateway connects:

Multiple VPCs
VPN connections (site-to-site)
Direct Connect Gateways (dedicated network link to on-premise)
Other Transit Gateways (even across Regions, via peering)

Analogy: VPC Peering is like every house in a neighborhood digging a private tunnel directly to every other house — expensive and chaotic to maintain. Transit Gateway is like a central roundabout — every house connects to the roundabout once, and the roundabout directs traffic to wherever it needs to go.

🎯 Interview/Cert scenario: "A company has 15 VPCs across different teams that all need to communicate with each other and with an on-premise data center. What's the best approach?" → Transit Gateway. VPC Peering would require dozens of individual connections and doesn't scale; Transit Gateway centralizes routing through one hub and also connects to on-premise via VPN/Direct Connect.

⚡ Quick Revision

Route Table Core

Destination = WHERE (the IP range) · Target = HOW (gateway/component used to get there)
Longest Prefix Match = most specific CIDR always wins
Same prefix length? Static route beats Propagated route

Route Propagation

Auto-learned routes from VGW/Transit Gateway via BGP — no manual entry needed, updates automatically

Blackhole Route

Route still exists, target no longer exists → traffic silently dropped
Common cause: NAT Gateway or Peering connection deleted
AWS flags it visibly as "Blackhole" in console

Managed Prefix List

Reusable named CIDR group, reference in many SGs/Route Tables
Update once → propagates everywhere it's referenced

Troubleshooting Order (memorize this sequence)

Route Table Association → 2. Longest Prefix Match → 3. Target Status → 4. Address Translation Path → 5. Security (SG + NACL)

VPC Flow Logs

Metadata only (not packet content) — source/dest IP+port, protocol, bytes, ACCEPT/REJECT
Destinations: CloudWatch Logs or S3 — forensic + security investigation tool

VPC Endpoints

Gateway Endpoint: S3 + DynamoDB only, free, route table based
Interface Endpoint: most other services, ENI + private IP, hourly cost
Private DNS on Interface Endpoint OVERRIDES default DNS resolution
Endpoint Policy = extra restriction layer on top of IAM

Transit Gateway

Central hub connecting many VPCs + VPN + Direct Connect
Solves the peering mesh scalability problem — one connection per VPC to the hub, not to every other VPC

💼 Interview Questions

Q1: What is Longest Prefix Match and why does it matter?
It's the rule AWS uses when multiple routes in a table could match a destination IP — the route with the most specific (longest) CIDR prefix always wins, regardless of when it was added. For example, a /24 route wins over a /16 route even if the /16 was configured first, as long as the destination IP falls within the /24 range.

Q2: What is a Blackhole route and how do you detect it?
A Blackhole route is a route whose target (like a NAT Gateway or Peering connection) no longer exists or is unavailable. Traffic matching that route gets silently dropped. AWS visually flags it as "Blackhole" directly in the Route Table console, making it easy to spot during troubleshooting.

Q3: A private EC2 instance suddenly lost internet access. Walk through your troubleshooting steps.
First check if the NAT Gateway is still available. Then verify the route table still points to the correct NAT Gateway ID and isn't marked Blackhole. Then check Security Groups and NACLs are still allowing the traffic in both directions. This systematic order — route table, target status, then security — quickly isolates whether it's a routing problem or a security rule problem.

Q4: What is the difference between a Gateway Endpoint and an Interface Endpoint?
Gateway Endpoints only support S3 and DynamoDB, work via a route table entry, and are free. Interface Endpoints support most other AWS services, work by creating an ENI with a private IP inside your subnet, have an hourly + data processing cost, and can be accessed from on-premises networks via VPN or Direct Connect — Gateway Endpoints cannot.

Q5: If both an S3 Gateway Endpoint and an S3 Interface Endpoint exist with Private DNS enabled, which is used?
The Interface Endpoint. Enabling Private DNS overrides the default S3 DNS resolution, so standard S3 endpoint calls resolve to the Interface Endpoint's private IP instead of routing through the Gateway Endpoint via the route table.

Q6: Why would a company use Transit Gateway instead of VPC Peering?
VPC Peering requires a direct connection between every pair of VPCs — this becomes unmanageable at scale (10 VPCs need 45 individual peering connections). Transit Gateway acts as a central hub — each VPC connects once to the Transit Gateway, which handles routing between all of them, and it also connects to on-premise networks via VPN or Direct Connect. Far more scalable for organizations with many VPCs.

Q7: What do VPC Flow Logs capture, and what don't they capture?
Flow Logs capture metadata about IP traffic — source/destination IP and port, protocol, packet/byte counts, timestamps, and whether traffic was accepted or rejected. They do NOT capture the actual content/payload of the traffic — you know that a connection happened and whether it was allowed, but not what data was inside it.

🔬 Practice Scenarios

Since this session is dense and interconnected, these scenarios are designed to be worked through slowly — the goal is confidence explaining "why," not just naming the term. Come back to these once the full VPC picture has settled after revisiting the lectures.

Longest Prefix Match drill: Given a route table with 10.0.0.0/16 → Local, 10.0.5.0/24 → pcx-abc (peering), and 0.0.0.0/0 → igw-xyz — for each of these destination IPs, state which route wins and why: 10.0.5.10, 10.0.9.10, 52.1.1.1.
Blackhole diagnosis: You delete a NAT Gateway to save cost. Two days later, private EC2 instances report they can't reach the internet. Walk through the full troubleshooting checklist and identify exactly where the problem would surface.
VPC Endpoint decision: Your app needs private access to S3, DynamoDB, and Secrets Manager without traffic touching the internet. Which endpoint type do you use for each service, and why?
Transit Gateway vs Peering: Your company has 3 VPCs today but plans to add 12 more over the next year, plus a VPN to the head office. Would you start with VPC Peering or Transit Gateway? Justify with the connection-count math.
Flow Logs investigation: You suspect someone is trying to SSH into your EC2 from an unauthorized IP. Describe exactly how you'd set up Flow Logs and what you'd filter for to confirm or rule this out.
Full lab (when ready): Build two VPCs, peer them, then replace the peering with a Transit Gateway once you have a third VPC. Add an S3 Gateway Endpoint to one VPC and confirm S3 traffic never leaves the AWS network by checking Flow Logs.

AWS VPC Extended | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS VPC — Your Private Network in the Cloud, Explained from the Ground Up

Tejas Shinkar — Tue, 28 Jul 2026 02:51:52 +0000

🌐 AWS Session 9 — Amazon VPC

Private Networking · Subnets · Internet Gateway · Route Tables · Security Groups · NACLs · NAT Gateway

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. VPC is the foundation everything else sits on — get this right and the rest of AWS networking clicks into place.

📋 Topics Covered

#	Topic	Type
1	Prerequisites — IP, Subnetting, Ports, DHCP, DNS	Concept
2	What is a VPC — Big Picture	Concept + Analogy
3	VPC as a Regional Resource	Concept + Interview
4	RFC 1918 — Private IP Ranges	Concept + Cert
5	CIDR Notation	Concept + Lab
6	Default VPC vs Custom VPC	Concept + Interview
7	VPC Tenancy — Shared vs Dedicated	Concept + Cert
8	Subnets — Public vs Private	Concept + Lab
9	Internet Gateway	Concept + Lab
10	Route Tables	Concept + Lab
11	Security Groups — Stateful Firewall	Concept + Interview
12	NACLs — Stateless Firewall	Concept + Interview
13	Stateful vs Stateless — Deep Explanation	Concept + Cert
14	NAT Gateway — Private Subnet Internet Access	Concept + Lab
15	How Traffic Flows — End to End	Concept + DevOps
16	Lab — Build a VPC from Scratch	Lab
17	Production VPC Architecture	DevOps
18	Interview Questions	Interview
19	Practice Tasks	Practice

The Big Picture First

Before any definition — here's what VPC actually does in plain terms.

When you sign up for AWS, you're sharing the same physical infrastructure as thousands of other companies. Without isolation, your servers could technically be reached by anyone else on that same hardware. VPC creates a logically isolated, private section of the AWS cloud that belongs only to you.

Think of it like this:

AWS is a massive co-working space with thousands of companies working in it. A VPC is your company's private floor — locked doors, your own network, your own rules. Other companies are in the same building but cannot reach your floor. Inside your floor, you decide which rooms (subnets) are open to visitors (public) and which are staff-only (private).

The whiteboard diagram says it perfectly:
"VPC as a service provides you an isolated environment by default. But you can build connectivity using various components for your organisation."

Prerequisites — Networking Foundations

You can't truly understand VPC without these five concepts. Don't skip them.

1. IP Address

An IP address is a unique logical address assigned to every device on a network so data knows where to go. Like a postal address — without it, packets have no destination.

IPv4 format:  192.168.1.100   (four numbers, 0-255, separated by dots)
IPv6 format:  2001:0db8:85a3::8a2e:0370:7334  (newer, longer, 128-bit)

In AWS VPC:
  Every EC2 instance gets a private IP  (e.g., 10.200.0.45)
  Internet-facing instances get a public IP too
  Elastic IP = static public IP that doesn't change on restart

2. Subnetting

A VPC gives you a large block of IP addresses (e.g., 10.200.0.0/16 = 65,536 IPs). You wouldn't hand all 65,536 addresses to random resources — you divide the VPC into subnets (smaller logical networks), each with its own range.

Why subnet?

Security — isolate databases from web servers
Organization — public-facing resources vs internal resources in separate segments
IP conservation — allocate only what each tier needs
AZ distribution — each subnet lives in one AZ

VPC: 10.200.0.0/16  (65,536 IPs total)
  ├── Public Subnet:   10.200.0.0/24   (256 IPs) — web servers, ALB
  ├── Private Subnet:  10.200.1.0/24   (256 IPs) — app servers
  └── Database Subnet: 10.200.2.0/24   (256 IPs) — RDS, databases

3. Ports & Protocols

An IP address gets you to the right device. A port gets you to the right service on that device.

Request: 10.200.0.45 : 80  →  reaches web server (Apache/Nginx)
Request: 10.200.0.45 : 22  →  reaches SSH daemon
Request: 10.200.0.45 : 3306 → reaches MySQL database

TCP vs UDP:

	TCP	UDP
Connection	Connection-oriented (handshake first)	Connectionless (just send)
Reliability	Guarantees delivery, retries on failure	No guarantee, no retry
Speed	Slower (overhead for reliability)	Faster (no overhead)
Use for	HTTP, SSH, databases, email	DNS, video streaming, gaming, VoIP

4. DHCP — Automatic IP Assignment

Without DHCP, every device joining a network would need manual IP configuration. DHCP automates this through the DORA process:

D — Discover   EC2 instance boots: "Is there a DHCP server?"
O — Offer      DHCP server: "Here, take IP 10.200.0.45"
R — Request    EC2: "Yes, I'll take that IP"
A — Acknowledge DHCP: "Confirmed. Here's your full config"

What DHCP delivers in AWS:
  1. IP Address        (e.g., 10.200.0.45)
  2. Subnet Mask       (e.g., 255.255.255.0)
  3. Default Gateway   (e.g., 10.200.0.1 — your VPC router)
  4. Primary DNS       (e.g., 169.254.169.253 — AWS DNS)
  5. Secondary DNS     (backup resolver)

In AWS, the VPC's built-in DHCP service handles all of this — EC2 instances get their config automatically at boot, no manual intervention.

5. DNS — The Internet's Phonebook

DNS translates domain names to IP addresses. Without it, you'd type 54.239.28.85 instead of amazon.com.

You type: www.amazon.com
      ↓
DNS resolver: "What's the IP for amazon.com?"
      ↓
DNS server returns: 54.239.28.85
      ↓
Your browser connects to 54.239.28.85

In AWS VPC:

Each VPC gets a built-in DNS resolver at 169.254.169.253 (or VPC base IP + 2)
EC2 hostnames like ip-10-200-0-45.ap-south-1.compute.internal are resolved automatically
Route 53 is AWS's managed DNS service — used for custom domain names

What is a VPC — With the Whiteboard Diagram Explained

From your class diagram, here's exactly what's happening when a packet travels between two EC2 instances across VPCs:

VPC 1 (Source)                              VPC 2 (Destination)
┌─────────────────────────────┐             ┌─────────────────────────────┐
│  EC2 - OS (App/App)         │             │  EC2 - OS (App/Dest)        │
│       ↕                     │             │       ↕                     │
│  NITRO CARD - ENI           │             │  NITRO CARD - ENI           │
│  (IP Address)               │             │  (IP Address)               │
│       ↕                     │             │       ↕                     │
│  SG Eval ←── Flow Logs      │             │  SG (Inbound Rules) → Drop/ │
│  Connection Tracking        │             │  Accept                     │
└──────────────┬──────────────┘             └──────────────▲──────────────┘
               │                                           │
               │    Encapsulate → Overlay Tunnel → Decapsulate
               └───────────────────────────────────────────┘

VPC is MAPPED to:
  ├── Subnet A  (NACL = Enforcement Point)
  └── Subnet B

Walking through this step by step:

EC2 OS generates a packet destined for the other VPC's IP
NITRO Card + ENI — AWS's custom hardware that handles networking. The ENI (Elastic Network Interface) is the virtual network card attached to the EC2
Security Group evaluation — before the packet leaves, SG rules are checked (outbound). Flow logs record the traffic. Connection tracking remembers this connection
Encapsulation — the packet is wrapped in another packet (overlay tunnel) and sent across the AWS backbone network
Decapsulation — at destination, the outer wrapper is removed
NACL enforcement — at the subnet boundary of the destination VPC, the NACL evaluates whether to allow or drop the inbound traffic
Security Group — at the destination ENI, inbound SG rules evaluated
EC2 OS receives the packet

VPC as a Regional Resource

AWS Account
    │
    ├── Region: ap-south-1 (Mumbai)
    │     ├── VPC 1 (10.0.0.0/16)
    │     │     ├── AZ: ap-south-1a → Subnet A (10.0.1.0/24)
    │     │     ├── AZ: ap-south-1b → Subnet B (10.0.2.0/24)
    │     │     └── AZ: ap-south-1c → Subnet C (10.0.3.0/24)
    │     │
    │     └── VPC 2 (10.200.0.0/16)  ← separate VPC, same region
    │
    └── Region: us-east-1 (N. Virginia)
          └── VPC 3 (10.1.0.0/16)  ← must be separate VPC for cross-region

Key rules:

A VPC belongs to exactly one Region — it cannot span Regions
A VPC can span multiple AZs within that Region via subnets
Cross-Region networking: use VPC Peering (direct 1:1) or Transit Gateway (hub-and-spoke for many VPCs)
Default limit: 5 VPCs per Region (soft limit, can request increase via Service Quotas)

🎯 Interview scenario: "A customer says they cannot create a new VPC." → Check if they've hit the 5 VPC per Region quota. Go to Service Quotas → request increase, or delete unused VPCs. Always check the specific Region they're trying to create in — limits are per-Region.

RFC 1918 — Private IP Ranges

Not all IP addresses are created equal. RFC 1918 (defined by IETF) reserves three ranges specifically for private networks — they are never routed on the public internet.

Range               CIDR Block      Total IPs      Common Use
─────────────────────────────────────────────────────────────
10.0.0.0 - 10.255.255.255   10.0.0.0/8      16,777,216    Large enterprise VPCs
172.16.0.0 - 172.31.255.255 172.16.0.0/12   1,048,576     Medium networks
192.168.0.0 - 192.168.255.255 192.168.0.0/16 65,536       Home routers, small offices

These are the only IP ranges you should use for VPC CIDR blocks in AWS.

Why? Because private IPs are not routable on the internet — two companies can both use 10.0.0.0/16 internally without conflict, since those IPs never appear on the public internet.

⚠️ Overlap trap: If two VPCs you want to peer have overlapping CIDR blocks (e.g., both use 10.0.0.0/16), you cannot peer them. Plan CIDR ranges carefully upfront in multi-VPC architectures.

CIDR Notation — Reading IP Ranges

CIDR (Classless Inter-Domain Routing) is how IP ranges are written. Understanding the / notation is essential.

10.200.0.0/16

10.200.0.0 = the starting IP address
/16        = 16 bits are fixed (the "network" part)
             remaining 32-16 = 16 bits are flexible (the "host" part)
             2^16 = 65,536 total IP addresses in this range

Quick reference:

CIDR	IPs Available	Use Case
/16	65,536	Entire VPC (large)
/24	256	One subnet
/28	16	Small subnet (minimum for AWS)
/32	1	Single specific IP (used in rules)
/0	All IPs	"Anywhere" (used in internet routes)

AWS reserves 5 IPs in every subnet:

10.200.0.0   → Network address
10.200.0.1   → VPC router (your default gateway)
10.200.0.2   → DNS resolver
10.200.0.3   → Reserved for future use
10.200.0.255 → Broadcast address

So a /24 subnet = 256 - 5 = 251 usable IPs

🎯 Cert tip: AWS always takes 5 IPs per subnet. Exam questions sometimes ask how many usable hosts are in a /28 subnet — answer is 16 - 5 = 11.

Default VPC vs Custom VPC

	Default VPC	Custom VPC
Created by	AWS automatically, one per Region	You create manually
CIDR	172.31.0.0/16 (always)	Your choice (RFC 1918)
Subnets	One public subnet per AZ (pre-created)	You design and create
Internet Gateway	Pre-attached	You create and attach
Route tables	Pre-configured with internet route	You configure
Public IPs	Auto-assigned to all instances	You choose per subnet
Use for	Quick learning, quick testing	Production, all real work

Should you delete the Default VPC?

Many organisations delete it or leave it deliberately empty. Why? If developers can accidentally launch resources into the Default VPC (which has public internet access by default), they may expose things that should be private. Custom VPCs enforce intentional network design.

🎯 Production standard: In real companies, the Default VPC is disabled or deleted. All production infrastructure runs in Custom VPCs with carefully designed subnet tiers, dedicated security groups per tier, and explicit routing.

VPC Tenancy — Shared vs Dedicated

Default Tenancy (Shared):
Your EC2 instances run on physical hardware shared with other AWS customers. The hypervisor (Nitro) provides full isolation — other customers cannot see your data — but the underlying physical server is shared.

Dedicated Tenancy:
Your EC2 instances run on physical hardware that no other AWS customer uses. The server is yours alone.

Default Tenancy:    [Your VM] [Other Customer VM] [Another VM]  ← same physical server
                    │          │                  │
                    └──────────┴──────────────────┘
                              Nitro hypervisor isolates them

Dedicated Tenancy:  [Your VM] [Your VM] [Your VM]  ← entire server is yours
                    No other customers on this hardware

When to use Dedicated:

Regulatory compliance requiring physical isolation (HIPAA, PCI-DSS)
Software licensing tied to physical cores/sockets
Security policy mandating dedicated hardware

Cost: Significantly more expensive than Default. Most workloads don't need it.

⚠️ One-way door: A VPC can be changed from Default → Dedicated tenancy. But Dedicated → Default cannot be changed back. If you want Default tenancy again, you must create a new VPC and migrate everything. Always think carefully before choosing Dedicated.

Subnets — Public vs Private

A subnet is a subdivision of your VPC — a smaller network within the larger network. What makes a subnet "public" or "private" isn't a setting you click — it's determined by the route table attached to it.

A subnet is PUBLIC if:
  → Its route table has a route: 0.0.0.0/0 → Internet Gateway
  → Instances can send/receive internet traffic
  → Instances should have a public IP (or Elastic IP)

A subnet is PRIVATE if:
  → Its route table has NO route to an Internet Gateway
  → No direct internet access
  → For private subnet internet access → needs NAT Gateway

Standard 3-tier architecture:

VPC: 10.200.0.0/16
│
├── Public Subnet: 10.200.0.0/24  (AZ-a)
│     Route: 0.0.0.0/0 → IGW
│     Resources: ALB, Bastion Host, NAT Gateway
│
├── Private Subnet: 10.200.1.0/24  (AZ-a)
│     Route: 0.0.0.0/0 → NAT Gateway
│     Resources: EC2 App Servers
│
└── Database Subnet: 10.200.2.0/24  (AZ-a)
      Route: local only
      Resources: RDS, ElastiCache
      No internet access at all

💡 Why keep databases in a separate subnet with no internet route? Even if someone compromises your app server, the database is in a completely separate network segment with no path to the internet. Lateral movement is constrained.

Internet Gateway (IGW)

An Internet Gateway is the door between your VPC and the public internet. Without it, nothing in your VPC can reach the internet and nothing from the internet can reach your VPC.

Internet
    │
Internet Gateway (IGW)
    │   ← attached to VPC (1:1 relationship)
    │
VPC

Key facts:

One VPC can have only one IGW
Horizontally scaled, redundant, and highly available — managed entirely by AWS
No bandwidth limits, no performance configuration needed
Free — you pay for data transfer, not the gateway itself
Performs NAT for public IPv4 — maps public IP to private IP for inbound traffic

Traffic flow for a public EC2 instance:

Internet user (203.0.113.5) → requests http://54.239.28.85 (EC2 public IP)
      ↓
Internet Gateway receives request
      ↓
IGW translates: 54.239.28.85 → 10.200.0.45 (EC2 private IP)
      ↓
Routes to EC2 instance in public subnet
      ↓
EC2 processes request, sends response
      ↓
Response goes back through IGW (private IP → public IP translation)
      ↓
Internet user receives response

Route Tables

A route table is a set of rules (routes) that determines where network traffic is directed. Every subnet must be associated with a route table — this is what makes it public or private.

How to read a route table:

Public Subnet Route Table:
Destination     Target
10.200.0.0/16   local         ← all VPC-internal traffic stays inside
0.0.0.0/0       igw-xxxxx     ← all other traffic goes to Internet Gateway

Private Subnet Route Table:
Destination     Target
10.200.0.0/16   local         ← VPC-internal traffic stays inside
0.0.0.0/0       nat-xxxxx     ← internet traffic goes through NAT Gateway
                              (NAT in public subnet bridges to IGW)

Database Subnet Route Table:
Destination     Target
10.200.0.0/16   local         ← only internal VPC traffic allowed
                              no 0.0.0.0/0 route = no internet at all

Most specific route wins:

Traffic to 10.200.1.50:
  Match: 10.200.0.0/16 → local  (/16 matches)
  No match for 0.0.0.0/0 needed
  → stays inside VPC ✅

Traffic to 8.8.8.8 (Google DNS):
  No match for 10.200.0.0/16  (8.8.8.8 is not in this range)
  Match: 0.0.0.0/0 → IGW  (catch-all)
  → goes to internet via IGW ✅

💡 One Main Route Table: Every VPC has a main route table (implicitly attached to all subnets by default). You create custom route tables and explicitly associate them with specific subnets to override the main table. Always use custom route tables per subnet tier — never rely on the main table for production routing.

Security Groups — Stateful Firewall

A Security Group is a stateful virtual firewall that controls traffic at the ENI level — attached to EC2 instances, RDS databases, and Load Balancers.

Analogy: The SG is the lock on an employee's cabin door. Only people on the approved list get in. And because it's stateful, if the employee walks out to get coffee (outbound), they're automatically let back in (return traffic) — no separate rule needed.

Key behaviours:

Default rules:
  Inbound:  DENY all  (nothing gets in unless you explicitly allow it)
  Outbound: ALLOW all (everything can leave by default)

Stateful = tracks connections:
  Outbound rule allows EC2 to call Google (port 443)
  → Return traffic from Google is automatically allowed
  → No inbound rule needed for the response

Important limitations:

Security Groups can only ALLOW — they cannot explicitly deny specific IPs
If you want to block a specific IP, you need a NACL (covered next)
A Security Group cannot be applied to a VPC — only to resources (ENIs)
Multiple SGs can be attached to one resource (rules are combined)
SGs can reference other SGs as source (e.g., allow traffic from SG-ALB)

NACLs — Stateless Firewall

A NACL (Network Access Control List) is a stateless firewall at the subnet boundary — it evaluates traffic entering and leaving a subnet before it even reaches any EC2 instance.

Analogy: The NACL is the security guard at the department entrance (subnet boundary). Before anyone enters the department, the guard checks them against a list. The guard doesn't care if someone is "already inside" — every time they pass through the door, they're checked again.

Internet traffic
      ↓
NACL Inbound Rules (evaluated at subnet entry)
      ↓  (if allowed)
EC2 Instance (Security Group evaluated at ENI)
      ↓  (response)
NACL Outbound Rules (evaluated at subnet exit)
      ↓  (if allowed)
Internet

NACL rules have numbered priorities — lower number = evaluated first:

Rule #  Protocol  Port    Source          Allow/Deny
100     TCP       80      0.0.0.0/0       ALLOW   ← HTTP allowed
110     TCP       443     0.0.0.0/0       ALLOW   ← HTTPS allowed
120     TCP       22      10.200.0.0/16   ALLOW   ← SSH from VPC only
*       ALL       ALL     0.0.0.0/0       DENY    ← default: deny everything else

The * rule at the bottom is the catch-all deny — always present, cannot be modified.

Stateful vs Stateless — The Critical Difference

This is one of the most tested concepts in AWS certs and a very common interview question.

	Security Group (Stateful)	NACL (Stateless)
Tracks connections?	✅ Yes	❌ No
Return traffic	Auto-allowed	Must explicitly allow
Applied at	ENI (instance level)	Subnet boundary
Rules	Allow only	Allow + Deny
Rule evaluation	All rules evaluated together	In order, first match wins
Scope	Per-resource	Per-subnet (all resources in subnet)
Default	Deny all inbound, Allow all outbound	Allow all (default NACL)

The stateful vs stateless scenario that always appears in exams:

EC2 in public subnet makes an outbound HTTPS call to api.example.com (port 443)

Security Group evaluation:
  Outbound rule: TCP 443 → ALLOW ✅
  Return traffic (inbound from api.example.com): AUTO-ALLOWED ✅
  (Stateful — tracks the connection, no explicit inbound rule needed)

NACL evaluation (outbound request):
  Outbound rule: TCP 443 → ALLOW ✅
  Return traffic comes back on EPHEMERAL PORT (1024-65535)
  NACL inbound rule needed: TCP 1024-65535 → ALLOW ✅
  (Stateless — doesn't remember the connection, must explicitly allow return traffic)

🎯 Classic exam question: "An EC2 can send requests but cannot receive responses. Security Group allows outbound TCP 443. NACL allows outbound TCP 443. What's wrong?" → The NACL is missing an inbound rule for ephemeral ports (1024-65535) for the return traffic. Security Groups don't need it (stateful). NACLs do (stateless).

Ephemeral ports: When your EC2 makes an outbound connection to port 443, the response comes back on a randomly assigned ephemeral port (1024-65535). NACLs need an explicit rule to allow this return range. Security Groups handle it automatically.

NAT Gateway — Private Subnet Internet Access

Private subnets have no direct path to the internet — that's intentional. But sometimes private instances need internet access outbound: to download software updates (yum update), reach external APIs, or pull container images.

NAT = Network Address Translation

A NAT Gateway sits in the public subnet and acts as a proxy — private instances send traffic to it, it forwards to the internet using its own public IP, gets the response, and sends it back to the private instance.

Analogy: A NAT Gateway is like a receptionist. Internal staff (private EC2) can ask the receptionist to call a vendor (internet service) on their behalf. The vendor speaks only to the receptionist (public IP), never knowing the internal staff member's internal extension (private IP). The receptionist takes the response and passes it back inside.

How it works — step by step:

Private EC2 (10.200.1.45) wants to run: yum update

Step 1: EC2 sends packet
  Source:      10.200.1.45 (private IP)
  Destination: 151.101.1.91 (yum repo server on internet)

Step 2: Route table sends to NAT Gateway
  Private subnet route: 0.0.0.0/0 → nat-gateway-id

Step 3: NAT Gateway translates
  Replaces source: 10.200.1.45 → 52.66.1.100 (NAT Gateway's Elastic IP)
  Forwards to internet via IGW

Step 4: Response comes back
  Source: 151.101.1.91 (yum server)
  Destination: 52.66.1.100 (NAT Gateway public IP)

Step 5: NAT Gateway translates back
  Looks up connection table: 52.66.1.100 → 10.200.1.45
  Sends response to private EC2

Step 6: Private EC2 receives yum update packages ✅
        Internet never knew the real private IP
        No unsolicited inbound from internet possible

Many-to-one NAT:
Multiple private instances can all share one NAT Gateway's public IP:

10.200.1.10 → outbound → 52.66.1.100 (NAT EIP)
10.200.1.11 → outbound → 52.66.1.100 (NAT EIP)
10.200.1.12 → outbound → 52.66.1.100 (NAT EIP)

The NAT Gateway tracks each connection in its translation table so return traffic goes to the correct private instance. The internet sees only one public IP.

NAT Gateway vs NAT Instance:

	NAT Gateway (AWS Managed)	NAT Instance (DIY on EC2)
Management	Fully managed by AWS	You manage (OS, updates)
Availability	Highly available within AZ	Single point of failure
Bandwidth	Up to 100 Gbps	Limited by instance type
Cost	Higher per hour	Lower per hour
Production use	✅ Always	❌ Avoid

✅ Always use NAT Gateway in production — never NAT Instance. The management overhead and single-point-of-failure risk of NAT Instance outweigh the cost savings.

💰 Cost note: NAT Gateway charges per hour + per GB of data processed. Put NAT Gateway in each AZ (not just one) for high availability — but this doubles the NAT cost. Trade-off between cost and availability.

End-to-End Traffic Flow — How It All Works Together

Scenario: A user in Mumbai opens your website hosted on a private EC2 behind an ALB.

User Browser (Mumbai)
      │
      │  HTTPS request to your domain
      ▼
Route 53 (DNS)
      │  resolves domain → ALB DNS name → ALB IP
      ▼
Internet Gateway (IGW)
      │  traffic enters VPC
      ▼
NACL (Public Subnet boundary)
      │  inbound rule: TCP 443 → ALLOW
      ▼
Application Load Balancer (in Public Subnet)
      │  SG-ALB: allows TCP 443 from 0.0.0.0/0
      │  ALB decrypts HTTPS, picks healthy EC2 target
      ▼
NACL (Private Subnet boundary)
      │  inbound rule: TCP 8080 from VPC CIDR → ALLOW
      ▼
EC2 App Server (in Private Subnet)
      │  SG-EC2: allows TCP 8080 from SG-ALB only
      │  App processes request, queries database
      ▼
RDS Database (in DB Subnet)
      │  SG-RDS: allows TCP 3306 from SG-EC2 only
      │  Returns query results
      ▼
Response travels back up the same path ↑

This is the standard 3-tier architecture deployed in every serious production environment.

Lab — Build a VPC from Scratch

This is exactly what you did in class. Walk through it again to make the steps click.

What You're Building

VPC: 10.200.0.0/16
  ├── Public Subnet: 10.200.0.0/24  (connected to IGW)
  ├── Private Subnet: 10.200.1.0/24 (connected to NAT GW)
  ├── Internet Gateway (attached to VPC)
  ├── NAT Gateway (in public subnet)
  ├── Public Route Table (0.0.0.0/0 → IGW)
  └── Private Route Table (0.0.0.0/0 → NAT GW)

Step 1 — Create VPC

VPC Console → Your VPCs → Create VPC

Resource:     VPC only (not "VPC and more")
Name tag:     lab-vpc
IPv4 CIDR:    10.200.0.0/16  (manual input)
IPv6 CIDR:    No
Tenancy:      Default

→ Create VPC

✅ Checkpoint: lab-vpc appears in Your VPCs list with state Available

Step 2 — Create Subnets

Public Subnet:

Subnets → Create subnet

VPC:              lab-vpc
Subnet name:      public-subnet-1
Availability Zone: ap-south-1a
IPv4 CIDR block:  10.200.0.0/24

→ Create subnet

Private Subnet:

Subnet name:      private-subnet-1
Availability Zone: ap-south-1b
IPv4 CIDR block:  10.200.1.0/24

💡 Put public and private subnets in different AZs — this improves availability. Even if one AZ has issues, the other tier keeps running.

✅ Checkpoint: Both subnets appear, status Available

Step 3 — Create Internet Gateway

Internet Gateways → Create internet gateway

Name tag: lab-igw

→ Create
→ After creation: Actions → Attach to VPC → select lab-vpc → Attach

⚠️ After creating an IGW, it shows "Detached" until you attach it to a VPC. This is the step most beginners forget — the IGW exists but does nothing until attached.

✅ Checkpoint: lab-igw shows state Attached to lab-vpc

Step 4 — Create Route Tables

Public Route Table:

Route Tables → Create route table
Name: public-rt
VPC: lab-vpc → Create

Edit Routes:
  Add route:
    Destination: 0.0.0.0/0
    Target: lab-igw
  Save routes

Subnet Associations → Edit subnet associations:
  Select: public-subnet-1
  Save

Private Route Table (add after NAT Gateway is created):

Route Tables → Create route table
Name: private-rt
VPC: lab-vpc → Create

(Add 0.0.0.0/0 → NAT Gateway after Step 5)

Subnet Associations → Edit subnet associations:
  Select: private-subnet-1
  Save

Step 5 — Create NAT Gateway

NAT Gateways → Create NAT gateway

Name:              lab-nat-gw
Subnet:            public-subnet-1  ← MUST be in public subnet
Connectivity type: Public
Elastic IP:        Allocate Elastic IP → Allocate

→ Create NAT gateway
Wait 2-3 minutes for state: Available

⚠️ NAT Gateway must be in the public subnet — it needs internet access via the IGW to forward traffic on behalf of private instances.

Now update private route table:

private-rt → Routes → Edit routes
Add route:
  Destination: 0.0.0.0/0
  Target: lab-nat-gw
Save routes

Step 6 — Launch EC2 Instances to Test

Public EC2:

EC2 → Launch Instance
Name: public-ec2
AMI: Amazon Linux 2023
Type: t2.micro
VPC: lab-vpc
Subnet: public-subnet-1
Auto-assign public IP: Enable
Security Group: allow SSH (22) from My IP + HTTP (80) from anywhere

Private EC2:

Name: private-ec2
VPC: lab-vpc
Subnet: private-subnet-1
Auto-assign public IP: Disable  ← no public IP
Security Group: allow SSH (22) from VPC CIDR (10.200.0.0/16)

Testing connectivity:

# SSH into public EC2 (has public IP)
ssh -i key.pem ec2-user@<public-ec2-public-ip>

# From public EC2, SSH into private EC2 (using private IP)
ssh -i key.pem ec2-user@<private-ec2-private-ip>

# From private EC2, test internet access via NAT Gateway
curl https://checkip.amazonaws.com
# Returns: NAT Gateway's Elastic IP (not the private EC2's IP) ✅

# Test outbound (yum update should work)
sudo yum update -y  # downloads via NAT Gateway ✅

🏢 Production VPC Architecture

This is what a real company's VPC looks like in AWS. Understanding this at the architecture level is what separates someone who knows AWS features from someone who can design AWS systems.

Region: ap-south-1 (Mumbai)
VPC: prod-vpc  10.0.0.0/16
│
├── AZ: ap-south-1a                    AZ: ap-south-1b
│   ├── Public Subnet  10.0.0.0/24    Public Subnet  10.0.10.0/24
│   │   ├── ALB node                  ├── ALB node
│   │   └── NAT Gateway               └── NAT Gateway
│   │
│   ├── Private App Subnet 10.0.1.0/24   Private App Subnet 10.0.11.0/24
│   │   └── EC2 (App Server)             └── EC2 (App Server)
│   │       ASG manages these             ASG manages these
│   │
│   └── Private DB Subnet 10.0.2.0/24    Private DB Subnet 10.0.12.0/24
│       └── RDS Primary                  └── RDS Standby (Multi-AZ)
│
├── Internet Gateway (single, attached to VPC)
├── Route 53 (DNS → ALB)
└── CloudFront (CDN → ALB or S3)

Security controls:
  SG-ALB:  HTTP/HTTPS from 0.0.0.0/0
  SG-EC2:  HTTP from SG-ALB only
  SG-RDS:  MySQL (3306) from SG-EC2 only
  NACL-Public:  Allow 80, 443, 1024-65535 inbound; Allow all outbound
  NACL-Private: Allow from VPC CIDR; No direct internet

Why this design?

Users only ever reach the ALB — EC2 and RDS are never exposed to internet
Even if app servers are compromised, no direct database access from internet
Multi-AZ for everything — if one AZ goes down, the other keeps serving
NAT Gateways in each AZ — private instances don't go cross-AZ for internet (reduces cost and latency)

⚡ Quick Revision

VPC FUNDAMENTALS
  Regional resource — one Region, spans multiple AZs via subnets
  Default limit: 5 VPCs per Region (soft limit)
  CIDR must be from RFC 1918 private ranges
  Overlapping CIDRs = cannot peer VPCs

SUBNET
  One AZ per subnet (subnet lives in one AZ)
  Public = route table has 0.0.0.0/0 → IGW
  Private = route table has 0.0.0.0/0 → NAT GW (or no internet route)
  AWS reserves 5 IPs per subnet

INTERNET GATEWAY
  One IGW per VPC (1:1 relationship)
  Must attach to VPC after creating
  Provides internet access for public subnets
  Performs NAT for public IPs

ROUTE TABLE
  Controls where traffic goes
  Most specific route wins
  Each subnet must have one associated route table
  Local route (VPC CIDR) always present, cannot delete

SECURITY GROUP (Stateful)
  Operates at ENI / instance level
  Tracks connections — return traffic auto-allowed
  Allow rules only (cannot deny specific IPs)
  Default: deny all inbound, allow all outbound

NACL (Stateless)
  Operates at subnet boundary
  Numbered rules, evaluated in order (lowest first)
  Must allow BOTH directions (including return/ephemeral ports 1024-65535)
  Allow + Deny rules
  Default NACL: allow all

NAT GATEWAY
  Must be in public subnet + has Elastic IP
  Private instances → NAT GW → IGW → Internet
  Outbound only from private instances (no unsolicited inbound)
  Many private IPs share one public IP
  Always use NAT Gateway (not NAT Instance) in production

DHCP (DORA)
  Discover → Offer → Request → Acknowledge
  Gives: IP, Subnet Mask, Default Gateway, DNS servers

STATEFUL vs STATELESS
  Stateful (SG):  tracks connection, return traffic auto-allowed
  Stateless (NACL): no memory, both directions need explicit rules

💼 Interview Questions

Q1: What is the difference between a Security Group and a NACL?
Security Groups are stateful firewalls at the ENI/instance level — they track connections so return traffic is automatically allowed, and they support Allow rules only. NACLs are stateless firewalls at the subnet boundary — they don't track connections so you need explicit rules for both directions (including ephemeral ports for return traffic), and they support both Allow and Deny rules.

Q2: A private EC2 instance can send requests to the internet but cannot receive responses. What's likely wrong?
The NACL for the private subnet is missing an inbound rule for ephemeral ports (1024-65535). The request goes out fine, but the response comes back on a dynamically assigned port in this range. The Security Group doesn't need a rule (stateful), but the NACL does (stateless). Add NACL inbound rule: TCP 1024-65535 from 0.0.0.0/0.

Q3: A customer says they cannot create a new VPC. What do you check first?
Check if they've hit the VPC quota for that Region — the default is 5 VPCs per Region. Go to Service Quotas and request an increase, or identify and delete unused VPCs. Also confirm they're checking the correct Region.

Q4: What makes a subnet "public" vs "private" in AWS?
Not a setting — it's determined by the route table. A subnet is public if its route table contains a route sending 0.0.0.0/0 traffic to an Internet Gateway. A subnet is private if it has no such route (or routes internet traffic through a NAT Gateway instead).

Q5: Why must a NAT Gateway be placed in a public subnet?
The NAT Gateway needs to be reachable from the internet to forward traffic on behalf of private instances. It must itself have a path to the Internet Gateway, which only exists in public subnets (via the route table). If it were in a private subnet, it would have no internet path and couldn't perform its function.

Q6: Can two VPCs with CIDR 10.0.0.0/16 be peered?
No. VPC Peering requires non-overlapping CIDR blocks. Both VPCs using 10.0.0.0/16 means their IP ranges are identical — routing would be ambiguous. You'd need to change one VPC's CIDR (which requires recreation) or use a different range from the start. This is why planning CIDR blocks upfront in multi-VPC architectures is critical.

Q7: What is the difference between an Internet Gateway and a NAT Gateway?
An Internet Gateway enables two-way communication between public subnet resources and the internet — inbound and outbound. A NAT Gateway enables outbound-only internet access for private subnet resources — it translates private IPs to its public IP for outbound traffic but blocks all unsolicited inbound connections from the internet.

🔬 Practice Tasks

Build from scratch (mandatory): Without referring to the lab steps, create a complete VPC: custom CIDR, one public and one private subnet in different AZs, IGW attached, NAT Gateway in public subnet, separate route tables per subnet with correct routes. Launch one EC2 in each subnet and verify: public EC2 reachable from internet, private EC2 reachable only from public EC2 via SSH, private EC2 can reach internet via NAT.
NACL test: Create a NACL that blocks HTTP (port 80) inbound to a public subnet but allows HTTPS (443). Launch an EC2 with Apache. Confirm HTTP is blocked but HTTPS works.
VPC Flow Logs: Enable VPC Flow Logs on your VPC. Generate some traffic (SSH, HTTP). Find the logs in CloudWatch and identify: source IP, destination IP, port, and whether traffic was accepted or rejected.
Stateless gotcha: Launch a private EC2 with a custom NACL that allows TCP 443 outbound. Try to reach an HTTPS endpoint. It will fail. Figure out why (missing ephemeral port inbound rule). Fix it and confirm it works.
Multi-AZ setup: Create subnets in 2 AZs, one public + one private in each AZ. Place a NAT Gateway in each public subnet. Configure private subnets to use the NAT Gateway in their own AZ. Test that private instances in each AZ reach the internet through their local NAT (check source IP).

AWS Session 9 — VPC Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS S3 — Versioning, Static Hosting, CORS, Object Lock & More

Tejas Shinkar — Sun, 26 Jul 2026 10:24:48 +0000

🗃️ AWS S3 — Extended Concepts

Versioning Deep Dive · Bucket Types · Static Hosting · CORS · Object Lock · Data Concepts

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session goes deeper into S3 — versioning behaviour, static website hosting, CORS, Object Lock, and important data concepts every cloud engineer should know.

📋 Topics Covered

#	Topic	Type
1	Bucket Versioning — Deep Dive	Concept + Cert
2	Does S3 Store Only Changed Bytes?	Concept + Interview
3	Suspend Versioning — What Actually Happens	Concept + Cert
4	General Purpose vs Directory Bucket	Concept + Cert
5	Storage Classes — Context Recap	Concept
6	OLTP vs OLAP — Data Concepts for Cloud Engineers	Concept + Interview
7	Static Website Hosting on S3	Concept + Lab
8	CORS — What It Is and Why S3 Needs It	Concept + Interview
9	S3 Object Lock — WORM Model	Concept + Cert
10	Interview Questions	Interview
11	Practice Tasks	Practice

Bucket Versioning — Deep Dive

You know what versioning does at a surface level — it keeps multiple versions of an object. But the behaviour in two specific scenarios is what most people get wrong, and both come up in exams and interviews.

Normal Upload Flow (Versioning Enabled)

Upload report.pdf           → Version ID: v1 (current)
Upload report.pdf again     → Version ID: v2 (current), v1 preserved
Upload report.pdf again     → Version ID: v3 (current), v1 + v2 preserved

GET report.pdf              → returns v3 (always returns current)
GET report.pdf?versionId=v1 → returns v1 specifically

Every upload with the same key creates a new version. The previous ones are never touched.

Delete Flow (The Tricky Part)

This is where most people get confused — and where interviewers test.

State: report.pdf has v1, v2, v3

DELETE report.pdf (standard delete, no versionId specified)
→ S3 does NOT delete anything
→ S3 adds a Delete Marker (a special object with its own Version ID)
→ GET report.pdf now returns 404 (marker hides the object)
→ But v1, v2, v3 still exist underneath
→ You're still paying storage for all three versions

Restoring a "deleted" object:

Delete the Delete Marker
→ Object reappears, GET report.pdf returns v3 again ✅

Permanently deleting an object and all versions:

Must explicitly delete each version by its Version ID:
DELETE report.pdf?versionId=v1
DELETE report.pdf?versionId=v2
DELETE report.pdf?versionId=v3
DELETE <delete-marker>?versionId=dm-id
→ Now the object is truly gone, zero storage consumed

⚠️ Cost trap: People enable versioning, never clean up old versions, and then wonder why their S3 bill is growing. Every version consumes storage. Fix: add a Lifecycle rule to expire non-current versions after X days.

Lifecycle rule → Expire non-current versions after 30 days
→ Old versions auto-deleted after 30 days
→ Current version always kept

Does S3 Store Only Changed Bytes?

A natural question — modern tools like git and incremental backups only store the difference (delta) between versions. Does S3 do the same?

No. S3 stores each version as a complete, separate object.

report.pdf v1 = 10 MB → stored in full (10 MB)
report.pdf v2 = 10.2 MB → stored in full (10.2 MB)  ← NOT just the 0.2 MB diff
report.pdf v3 = 10.1 MB → stored in full (10.1 MB)

Total storage consumed: 30.3 MB (not 10.2 MB)

Each version gets its own Version ID and is managed independently — you can access, copy, restore, or delete any version directly.

Implication for cost planning:

Frequently updated large files + versioning = storage costs multiply fast
For 100 uploads/day of a 50 MB file: 100 × 50 MB = 5 GB/day extra storage
Always pair versioning with Lifecycle rules that expire old versions

🎯 Interview tip: "Does S3 versioning use delta storage like git?" — No. Every version is a full copy. This is why Lifecycle policies to expire non-current versions are essential in production.

Suspend Versioning — What Actually Happens

Once you enable versioning on a bucket, you cannot fully turn it off — you can only suspend it. This is by design: AWS won't let you accidentally lose version history.

Behaviour After Suspension

Before suspension: v1, v2, v3 exist for report.pdf

Versioning suspended.

New upload: report.pdf v4
→ Gets Version ID: null (not a real UUID)
→ If another null version existed, it gets overwritten
→ v1, v2, v3 still safely preserved

State now:
  report.pdf  →  null version (v4 upload)
                 v3 (preserved)
                 v2 (preserved)
                 v1 (preserved)

Three states a bucket can be in:

State	New uploads	Existing versions
Unversioned (never enabled)	No Version ID	N/A
Enabled	Full Version ID (UUID)	All preserved
Suspended	Version ID: `null`	All preserved

Key rules to remember:

Versioning: once enabled, can never go back to "Unversioned"
Suspended → Enabled: you can re-enable. All existing versions preserved, new uploads get UUIDs again
Null version uploads: if you upload again while suspended, the previous null version is overwritten (but all UUID versions stay untouched)

🎯 Cert scenario: "A bucket has versioning enabled. The team wants to stop creating new versions to save costs but keep existing ones. What should they do?" → Suspend versioning. New uploads will use null version ID and won't accumulate new versions, but all existing versions are retained.

General Purpose Bucket vs Directory Bucket

Historically S3 had one type of bucket. AWS introduced Directory Buckets for a specific high-performance use case. Knowing the difference matters for both certs and architecture decisions.

General Purpose Bucket

The standard S3 bucket you've been using in all labs. Think of it as a general warehouse — stores anything, supports all features, multi-AZ redundancy by default.

Features:          ALL S3 features supported
AZs:               Multi-AZ (data replicated across 3+ AZs in a Region)
Storage classes:   All classes (Standard, IA, Glacier, etc.)
Versioning:        ✅ Supported
Lifecycle rules:   ✅ Supported
Object Lock:       ✅ Supported
Replication:       ✅ Supported
Use for:           Everything — backups, web assets, data lakes, logs

Directory Bucket

Purpose-built for high-performance, latency-sensitive workloads — think machine learning training data, real-time analytics, high-frequency log ingestion.

Think of it as an express warehouse inside one building — extremely fast, but only within a single AZ and stripped of most management features.

Features:          Limited to core operations
AZs:               Single AZ only (Express One Zone storage class)
Storage classes:   Only Express One Zone
Versioning:        ❌ Not supported
Lifecycle rules:   ❌ Not supported
Object Lock:       ❌ Not supported
Performance:       10x faster than General Purpose for small objects
Use for:           ML training, real-time data processing, low-latency apps

Decision guide:

Need all S3 features (versioning, lifecycle, replication)?  → General Purpose
Need maximum speed for ML/analytics inside one AZ?         → Directory Bucket
Need durability across multiple AZs?                       → General Purpose
Need sub-10ms consistent latency at massive scale?         → Directory Bucket

⚠️ Directory Buckets are in a single AZ — if that AZ has an outage, your data is temporarily unavailable. Only use them for workloads where performance beats durability requirements.

Storage Classes — Context Recap

Covered in depth last session. Quick mental model for reference:

How often will this data be accessed?

Daily/Active         → Standard
Unknown pattern      → Intelligent-Tiering (auto-manages itself)
Once a month         → Standard-IA (retrieval fee applies)
Once a quarter       → Glacier Instant Retrieval
Once a year          → Glacier Flexible (minutes-to-hours retrieval)
Almost never         → Glacier Deep Archive (cheapest, up to 12hr retrieval)
One AZ is fine + fast? → Express One Zone (Directory Bucket only)

Lifecycle rules automate the transitions — set it once, S3 moves objects down the chain automatically as they age. Always pair versioning with lifecycle rules that expire old versions.

OLTP vs OLAP — Data Concepts Every Cloud Engineer Should Know

This pair of concepts comes up when you're designing cloud architectures involving databases and data pipelines — which is constantly in DevOps and cloud roles.

OLTP — Online Transaction Processing

What it does: Handles day-to-day, real-time business operations — individual transactions, fast reads and writes, high concurrency.

Characteristics:

Short, fast queries (milliseconds)
Many concurrent users
Writes and reads constantly
Data is current and operational
Optimized for INSERT, UPDATE, DELETE

Examples in real life: A customer places an order, a payment is processed, a hotel room is booked — each of these is one OLTP transaction.

AWS services: RDS (MySQL, PostgreSQL), Aurora, DynamoDB

User clicks "Buy Now" on your site
      ↓
OLTP (RDS/Aurora)
      ↓
INSERT INTO orders VALUES (user_id, item_id, price, timestamp)
UPDATE inventory SET stock = stock - 1 WHERE item_id = 123
      ↓
Completed in milliseconds ✅

OLAP — Online Analytical Processing

What it does: Analyses large historical datasets to generate reports, trends, and business insights. Not for real-time transactions — for answering questions like "what were our top-selling products last quarter across all regions?"

Characteristics:

Complex queries across massive datasets (millions/billions of rows)
Read-heavy (rarely writes)
Aggregations, joins, GROUP BY across large tables
Historical data, not real-time
Optimized for fast analytical reads (columnar storage)

Examples in real life: Monthly sales report, customer churn analysis, inventory trend forecast.

AWS services: Amazon Redshift (data warehouse), Amazon Athena (query S3 directly with SQL)

Business analyst asks: "Show me total revenue by product category for 2025"
      ↓
OLAP (Redshift/Athena)
      ↓
SELECT category, SUM(revenue)
FROM sales_fact
WHERE year = 2025
GROUP BY category
      ↓
Scans billions of rows, returns in seconds ✅ (would kill an OLTP database)

Why Both Matter Together

In real architectures, you use both — OLTP for operations, OLAP for analysis:

Customer transactions → RDS (OLTP) → ETL pipeline → Redshift (OLAP)
                                           ↑
                               Runs nightly, loads transformed data
                               Business reports run against Redshift
                               RDS never touched by analytical queries

Why separate them? Running a heavy analytical query against your OLTP database would lock tables, slow down real user transactions, and potentially take the site down. OLAP databases are architecturally different — columnar storage, massively parallel processing — built specifically for large scans.

Quick Comparison

	OLTP	OLAP
Purpose	Real-time transactions	Historical analysis
Query type	Short, simple	Long, complex aggregations
Data volume	Thousands of rows	Billions of rows
Users	Many concurrent end users	Analysts, BI tools
Writes	Frequent	Rare (batch loads)
AWS services	RDS, Aurora, DynamoDB	Redshift, Athena
Storage	Row-oriented	Columnar (faster scans)

🎯 Interview context: If asked "how would you design a reporting system that doesn't slow down production?" — separate OLTP (RDS for transactions) from OLAP (Redshift/Athena for analytics), connected by an ETL pipeline. This pattern shows architectural thinking, not just tool knowledge.

Static Website Hosting on S3

S3 can serve a complete website — HTML, CSS, JavaScript, images — directly from a bucket, without any server.

Analogy: Normally a website needs a server running 24/7 waiting for visitors. Static hosting on S3 is like pinning your brochure to a public notice board — it's always there, no one needs to "run" it, and millions of people can read it simultaneously.

What "Static" Means

Static = files that are the same for every visitor. The server doesn't generate content dynamically per user.

✅ Static (S3 can serve):        ❌ Not static (S3 cannot serve):
   HTML files                       Node.js / Express apps
   CSS stylesheets                  PHP backend
   JavaScript (React, Vue)          Python Flask/Django
   Images, fonts, PDFs              Database queries
   Pre-built React apps             Server-side rendering

A React or Vue app that calls an API is fine — the frontend is static files, the API runs on EC2/Lambda elsewhere.

Step-by-Step Setup

Step 1: Create bucket
  Name: your-site-name.com (can match your domain)
  Region: us-east-1 (or closest to users)
  Block Public Access: DISABLE all 4 settings for public site

Step 2: Enable Static Website Hosting
  Bucket → Properties → Static website hosting → Enable
  Index document: index.html
  Error document: error.html (optional)
  Save → note the Website Endpoint URL

Step 3: Upload your files
  Upload index.html, styles.css, app.js, images/
  Make sure all files are present

Step 4: Add Bucket Policy (allow public read)
  Bucket → Permissions → Bucket Policy → Edit → paste:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "PublicReadGetObject",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::your-bucket-name/*"
  }]
}

Step 5: Test
  Open the S3 Website Endpoint in browser
  http://your-bucket-name.s3-website-ap-south-1.amazonaws.com
  → Your site loads ✅

S3 Website URL vs S3 Object URL

These look similar but behave differently:

	S3 Object URL	S3 Website Endpoint
Format	`bucket.s3.region.amazonaws.com/key`	`bucket.s3-website-region.amazonaws.com`
Returns	The raw file	Serves index.html for `/`, error.html for 404
HTTPS	✅	❌ HTTP only
CloudFront compatible	✅	✅

✅ Production setup: Use CloudFront in front of S3 static hosting to get HTTPS, custom domain, and CDN caching. S3 alone gives you HTTP only with an ugly URL.

Best for: Personal portfolio, documentation site, React/Vue/Angular SPA, company landing page — anything without server-side logic.

CORS — Cross-Origin Resource Sharing

The Problem CORS Solves

Browsers enforce a security rule called the Same-Origin Policy — a page loaded from domain-a.com cannot make JavaScript requests to domain-b.com unless explicitly permitted.

Analogy: You work at Company A and want to access files in Company B's building. Company B's security guard (the browser's CORS check) checks whether Company A employees are on the approved visitor list before letting you in.

Where This Appears with S3

Your frontend: https://myapp.com  (served from EC2 or elsewhere)
Your S3 bucket: https://my-assets.s3.amazonaws.com

User visits myapp.com, JavaScript runs:
  fetch('https://my-assets.s3.amazonaws.com/data.json')
  → Browser BLOCKS this request ❌
  → "Cross-origin request blocked"

The browser blocked it because myapp.com ≠ my-assets.s3.amazonaws.com — different origins.

Fixing It — CORS Configuration on S3

You add a CORS rule to the S3 bucket that tells browsers: "Requests from myapp.com are approved."

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST"],
    "AllowedOrigins": ["https://myapp.com"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

Breaking it down:

Field	Meaning
`AllowedOrigins`	Which domains can make requests — `["https://myapp.com"]` or `["*"]` for all
`AllowedMethods`	Which HTTP methods are allowed — GET, PUT, POST, DELETE, HEAD
`AllowedHeaders`	Which request headers are allowed — `["*"]` means all
`ExposeHeaders`	Which response headers the browser can read
`MaxAgeSeconds`	How long the browser can cache the CORS preflight response

Bucket Policy vs CORS — Key Distinction

This trips people up:

Bucket Policy: answers "WHO can access this bucket?"
  → Applies to: all clients (CLI, SDK, browser, curl)
  → Example: "Only my EC2 role can GetObject"

CORS: answers "Which BROWSER ORIGINS can use that access?"
  → Applies to: only browser-based JavaScript requests
  → Example: "myapp.com's JavaScript is allowed to fetch from this bucket"

Both must allow the request for a browser to successfully access S3:

Bucket Policy says the user/origin is allowed
CORS says the browser origin is allowed

🎯 Interview tip: "What's the difference between a bucket policy and CORS in S3?" — Bucket policy controls who can access (any client). CORS controls which browser origins can access (browser JS only). They're complementary, not interchangeable.

S3 Object Lock — WORM Model

What is WORM?

WORM = Write Once, Read Many — once data is written, it cannot be modified or deleted for a defined period. This is a compliance requirement in industries like finance, healthcare, and legal.

Analogy: Object Lock is like writing in permanent marker on a whiteboard that's been laminated — it can be read by anyone, but nobody can erase or change what's written during the protection period.

Two Retention Modes

Governance Mode

Objects cannot be modified or deleted by regular users
Authorized IAM users with s3:BypassGovernanceRetention permission can bypass the lock
Use for: internal policies where administrators may occasionally need override ability

Regular user: DELETE object → ❌ Blocked
Admin with bypass permission: DELETE object → ✅ Allowed (with override header)

Compliance Mode

Objects cannot be modified or deleted by anyone — including root user and AWS Support
Once set, the retention period cannot be shortened
Use for: regulated industries (SEC Rule 17a-4, HIPAA, FINRA) where even admins cannot bypass

Regular user: DELETE object → ❌ Blocked
Admin:        DELETE object → ❌ Blocked
Root user:    DELETE object → ❌ Blocked
AWS Support:  Cannot help delete → ❌ Blocked

⚠️ Compliance Mode is irreversible for the retention period. Think carefully before setting it — if you set a 10-year retention in Compliance Mode, that data cannot be deleted for 10 years by anyone.

Two Lock Mechanisms

Retention Period — protects the object version until a specific date

Object uploaded with retention: 2026-01-01 to 2036-01-01 (10 years)
→ Cannot be deleted or modified until 2036-01-01
→ After that date, normal rules apply

Legal Hold — protects indefinitely until explicitly removed (no date)

Object placed under Legal Hold
→ Cannot be deleted until Legal Hold is removed
→ No expiry date — remains until someone with s3:PutObjectLegalHold removes it
→ Used during litigation: "preserve everything related to this case"

Requirements & Use Cases

Requirements to enable Object Lock:

✅ Must enable at bucket creation (cannot enable on existing bucket)
✅ Versioning is automatically enabled and required
✅ Cannot disable versioning once Object Lock is active

Common use cases:

Industry	Requirement	Mode
Finance	SEC 17a-4: emails and records for 7 years	Compliance
Healthcare	HIPAA: patient records retention	Compliance
Legal	Litigation hold on documents	Legal Hold
Backup	Ransomware-proof backups	Governance
Government	Regulatory audit records	Compliance

🎯 Cert scenario: "A financial company needs to store trade records for 7 years and ensure not even admins can delete them." → S3 Object Lock in Compliance Mode with a 7-year retention period. Governance Mode would be wrong here because admins could bypass it.

Object Lock vs Versioning vs Delete Marker

Versioning alone:
  DELETE → adds delete marker, data still recoverable

Versioning + Object Lock (Governance):
  DELETE → blocked for regular users, admins can bypass
  Object still exists until retention period ends

Versioning + Object Lock (Compliance):
  DELETE → blocked for everyone, no bypass possible
  Object guaranteed to exist until retention date

⚡ Quick Revision

VERSIONING
  Upload same key → new version created (old preserved)
  Standard delete → adds Delete Marker (data not gone)
  Permanent delete → must delete each version by Version ID
  Each version = full copy, not delta → watch storage costs
  Fix: Lifecycle rule to expire non-current versions

SUSPEND VERSIONING
  Cannot fully disable once enabled — only suspend
  New uploads after suspension → null Version ID
  Existing versions → all preserved, untouched
  Re-enable → new uploads get UUIDs again

BUCKET TYPES
  General Purpose = all features, multi-AZ, everyday use
  Directory Bucket = single AZ, 10x faster, no versioning/lifecycle/lock

OLTP vs OLAP
  OLTP → transactions, fast R/W, RDS/Aurora/DynamoDB
  OLAP → analytics, large scans, Redshift/Athena
  Separate them in architecture — never run analytics on OLTP DB

STATIC HOSTING
  S3 serves HTML/CSS/JS/images directly
  Cannot run server-side code (Node, PHP, Python)
  Must disable Block Public Access + add public read policy
  Add CloudFront for HTTPS + custom domain in production

CORS
  Bucket Policy = WHO can access (all clients)
  CORS = WHICH BROWSER ORIGINS can access (browsers only)
  Both must allow request for browser JS to work
  Configure in bucket's Permissions → Cross-origin resource sharing

OBJECT LOCK (WORM)
  Must enable at bucket creation
  Requires versioning (auto-enabled)
  Governance = lock with admin bypass
  Compliance = lock with NO bypass (not even root)
  Retention Period = until specific date
  Legal Hold = indefinite until removed

💼 Interview Questions

Q1: What happens when you delete an object in a versioned S3 bucket?
S3 does not delete the actual data. Instead, it adds a Delete Marker — a special placeholder with its own Version ID. The object appears gone (GET returns 404), but all versions still exist and are still billed. To permanently delete, you must explicitly delete each version by its Version ID and remove the delete marker.

Q2: Does S3 versioning use delta/incremental storage like git?
No. Each version is stored as a complete, independent object with its own Version ID. Uploading a 10 MB file 10 times creates 100 MB of stored data, not 10 MB plus incremental diffs. This makes Lifecycle rules to expire non-current versions essential for cost control.

Q3: What is the difference between Governance Mode and Compliance Mode in S3 Object Lock?
Both prevent object modification and deletion. Governance Mode allows authorized IAM users with the bypass permission to override the lock — useful for internal policies where administrators need flexibility. Compliance Mode cannot be bypassed by anyone, including root user and AWS — used for strict regulatory requirements where data immutability must be guaranteed.

Q4: When does S3 CORS apply and when does the bucket policy apply?
The bucket policy applies to all access — CLI, SDK, curl, and browser — and controls who is allowed to make requests. CORS applies only to browser-based JavaScript requests and controls which origins (domains) browsers are allowed to make requests from. Both must permit the request for a browser to successfully access S3.

Q5: What is the difference between a General Purpose and Directory Bucket?
General Purpose is the standard S3 bucket — supports all features (versioning, lifecycle, object lock, all storage classes), data is replicated across multiple AZs, suitable for all workloads. Directory Bucket is purpose-built for high-performance single-AZ workloads — up to 10x faster for small objects, but supports only Express One Zone storage class, no versioning, no lifecycle rules, no object lock.

Q6: Can you disable versioning after enabling it on a bucket?
No. Once versioning is enabled, it can only be suspended — never fully disabled. Suspension stops new versions from being created (new uploads get null version ID) but all existing versions are permanently preserved. You can re-enable versioning at any time.

Q7: What is the difference between OLTP and OLAP? Give AWS examples.
OLTP handles real-time operational transactions — short, fast read/write queries, high concurrency, current data. AWS: RDS, Aurora, DynamoDB. OLAP handles large-scale analytical queries across historical data — complex aggregations, columnar storage, read-heavy. AWS: Redshift, Athena. They're always kept separate in architecture because analytical queries would overwhelm and destabilize a transactional database.

🔬 Practice Tasks

Versioning lab: Enable versioning on a bucket. Upload test.txt. Upload a modified version with the same name. List all versions via CLI (aws s3api list-object-versions). Delete the object via Console. Confirm the delete marker was created. Restore the object by deleting the delete marker. Then permanently delete all versions.
Lifecycle + Versioning: Create a bucket with versioning enabled. Add a Lifecycle rule to expire non-current versions after 7 days. Upload the same file 5 times. Verify all 5 versions appear. Wait (or check back after 7 days) to confirm old versions are auto-expired.
Static website: Create a bucket, disable Block Public Access, enable static website hosting, upload a simple index.html and error.html, add the public read bucket policy, and access the S3 website endpoint in your browser.
CORS lab: Create a bucket with an image. Try to fetch it via JavaScript in your browser's console from a different origin — observe the CORS error. Add a CORS rule allowing * as origin. Retry and confirm it succeeds.
Object Lock: Create a new bucket with Object Lock enabled (Governance Mode). Upload a file with a 1-day retention period. Try to delete it — confirm it's blocked. Check what happens when you try to overwrite it.
Write the policy: Without notes, write a bucket policy that: (a) allows public GET on all objects, (b) denies all HTTP (non-HTTPS) requests, and (c) allows a specific IAM role to PUT objects.

AWS S3 Extended Concepts | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS S3 Deep Dive — Objects, Encryption, Bucket Policies & Everything In Between

Tejas Shinkar — Mon, 20 Jul 2026 13:27:36 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session goes deep into S3 — the most widely used AWS service and a guaranteed topic in every AWS certification and DevOps interview.

📋 Topics Covered

#	Topic	Type
1	Why S3 Matters + Quick Recap	Concept
2	JSON & YAML — Why They Matter in AWS	Concept + Interview
3	API Calls in AWS (GET, PUT, DELETE)	Concept + DevOps
4	Region Selection — Why It Matters for S3	Concept + Cert
5	S3 Buckets — Rules & Constraints	Concept + Lab
6	S3 Objects — What They Actually Are	Concept + Cert
7	S3 Object URL Structure	Concept + Lab
8	S3 Storage Classes	Concept + Cert
9	S3 Versioning	Concept + Lab
10	S3 Encryption — SSE-S3, SSE-KMS, SSE-C, CSE	Concept + Cert
11	S3 Bucket Keys — KMS Cost Optimization	Concept + Cert
12	Bucket Policies — Ownership & ACL Settings	Concept + Interview
13	S3 Access Control — Policies, ACLs, Public Access	Concept + Cert
14	Explicit Deny — The Override Rule	Concept + Interview
15	AWS CloudTrail + S3 Audit Logging	Concept + DevOps
16	VPC — Brief Introduction	Concept
17	S3 CLI Commands	Practical
18	boto3 — S3 with Python	Practical
19	Interview Questions	Interview
20	Practice Tasks	Practice

Why S3 Matters

S3 (Simple Storage Service) is arguably the most important AWS service to understand — it appears in almost every architecture, every certification, and every DevOps workflow. Backups, static websites, data lakes, CI/CD artifacts, ML training data, log archives — S3 underpins all of it.

Before diving into encryption and policies, two foundational things from this session need to be covered first: JSON/YAML code and API calls — because everything in S3 (and AWS) is built on top of these.

JSON & YAML — Why Interviewers Ask You to Write Them

In real organizations, nobody clicks through the AWS Console to create resources. Everything is written as code — JSON or YAML — and executed via CLI, CloudFormation, or Terraform. Interviewers increasingly ask candidates to write infrastructure code on the spot (sometimes called "vibe coding").

JSON (JavaScript Object Notation):

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "MyEC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": "ami-0abcdef1234567890",
        "InstanceType": "t2.micro",
        "KeyName": "my-key-pair",
        "Tags": [
          { "Key": "Name", "Value": "MyServer" }
        ]
      }
    }
  }
}

YAML (Yet Another Markup Language) — same thing, cleaner to read:

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-0abcdef1234567890
      InstanceType: t2.micro
      KeyName: my-key-pair
      Tags:
        - Key: Name
          Value: MyServer

Key differences:

	JSON	YAML
Syntax	Curly braces `{}`, quotes `""`, commas	Indentation, colons `:`, cleaner
Readability	Verbose, harder to read	Clean, easier to write by hand
Comments	❌ Not supported	✅ Supported with `#`
Used in AWS for	API responses, Lambda, IAM policies	CloudFormation, SAM, CodePipeline
Error-prone	Missing comma or brace	Wrong indentation

S3 Bucket Policy in JSON (what you'll write in real interviews):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPublicRead",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket-name/*"
    }
  ]
}

IAM Policy in YAML (CloudFormation):

Type: AWS::IAM::Policy
Properties:
  PolicyName: S3ReadPolicy
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - s3:GetObject
          - s3:ListBucket
        Resource:
          - arn:aws:s3:::my-bucket
          - arn:aws:s3:::my-bucket/*

🎯 Interview tip: Practice writing a basic S3 bucket policy and EC2 IAM role in both JSON and YAML without looking things up. Interviewers genuinely ask for this now.

API Calls in AWS — How Everything Actually Works

Every click in the AWS Console, every CLI command, every boto3 call — all of it is ultimately an HTTP API call under the hood. Understanding this is essential for DevOps and automation work.

AWS uses standard HTTP methods:

Method	Purpose	S3 Example
`GET`	Read / retrieve data	Download an object, list buckets
`PUT`	Create or replace	Upload an object, create a bucket
`POST`	Create or trigger	Initiate multipart upload
`DELETE`	Remove	Delete an object or bucket
`HEAD`	Get metadata only	Check if object exists without downloading

EC2 → S3 interaction via API:

EC2 Instance
    │
    │  PUT /my-bucket/backup.zip  (upload)
    │  GET /my-bucket/config.json (download)
    │  DELETE /my-bucket/old.log  (delete)
    ▼
S3 API Endpoint: https://s3.ap-south-1.amazonaws.com
    │
    ▼
S3 Bucket (my-bucket)

In practice, your application on EC2 uses the AWS SDK (boto3 in Python, or the Java/Node SDK) which converts your code calls into these signed HTTP API requests automatically.

AWS Signature Version 4 — all API calls must be signed with your credentials so AWS can verify identity. The SDK handles this for you when using IAM roles (recommended) or access keys (not recommended).

🎯 DevOps relevance: When you write a Python script with boto3 to copy files to S3, you're making PUT API calls. When your Lambda function reads from DynamoDB, it's making GET API calls. The console is just a visual wrapper over the same APIs.

Region Selection — Why It Matters for S3

S3 is a global service in name, but buckets are created in a specific Region. Choosing the right region affects:

Factor	Impact on S3
Performance	Store data in Region closest to your users/EC2 → lower latency
Cost	S3 pricing varies by Region (US East is typically cheapest)
Data Transfer	Transferring data between Regions costs money — keep S3 and EC2 in same Region
Compliance / Regulatory	GDPR (EU), RBI (India) may legally require data in specific regions
Disaster Recovery	Cross-Region Replication (CRR) copies to another Region for DR

💡 Practical rule: Always create your S3 bucket in the same Region as the EC2 instances or services that will use it. Cross-region data transfer is charged per GB and adds latency.

S3 Buckets — Rules & Constraints

A bucket is the container for your objects in S3. Think of it like a top-level folder — except with globally unique naming rules and no nesting of buckets inside buckets.

Bucket Naming Rules

✅ Must be globally unique — no two buckets in the world can share a name
✅ 3 to 63 characters long
✅ Lowercase letters, numbers, and hyphens only
✅ Must start with a letter or number
✅ Must not start with "xn--" (reserved)
✅ Must not end with "-s3alias"
❌ No uppercase letters
❌ No underscores
❌ No IP address format (e.g., 192.168.1.1)
❌ No spaces

Good names: tejas-dev-backups, company-prod-logs-2026, my-static-site-assets

Bad names: My_Bucket, 123.456.789.0, TejasBucket

Key Bucket Facts

One Region per bucket — created in a specific Region, stays there
No size limit on total bucket storage — scales infinitely
5 TB max per individual object — single file can't exceed 5TB
Soft limit of 100 buckets per account (can request increase to 1000)
No nested buckets — you cannot put a bucket inside a bucket
Bucket names in URLs — because names are globally unique, they're used directly in URLs

S3 Objects — What They Actually Are

An object is any file stored in S3. Unlike a traditional file system, S3 is flat — there are no real folders (just prefixes that look like folders).

Object Components

Component	Description
Key	The full "path" of the object. e.g., `reports/2026/january/sales.csv`
Value	The actual file content (bytes)
Version ID	Unique ID if versioning is enabled
Metadata	Key-value pairs about the object (content-type, custom tags)
ETag	MD5 hash of the object (used to verify integrity)
Storage Class	Standard, IA, Glacier, etc.

The "Folder" Illusion

Bucket: my-data
Objects stored as:
  reports/2026/sales.csv    ← key includes the "path"
  reports/2026/revenue.csv
  images/logo.png
  images/banner.jpg
  config.json

There are NO real folders. "reports/2026/" is just a prefix.
The console shows them as folders for readability — but it's flat storage underneath.

Object Size Limits

Upload type	Size limit	When to use
Single PUT upload	Up to 5 GB	Small to medium files
Multipart upload	Up to 5 TB	Files larger than 100 MB (recommended)
Minimum part size	5 MB	For multipart upload chunks

✅ Best practice: Use multipart upload for any file over 100 MB. It's faster (parallel chunks), more reliable (resume on failure), and required above 5 GB.

S3 Object URL Structure

Every object in S3 has a predictable, deterministic URL. Understanding the format is important for linking to files, debugging access issues, and building applications.

URL Formats

Path-style URL (older, being deprecated):

https://s3.<region>.amazonaws.com/<bucket-name>/<object-key>
https://s3.ap-south-1.amazonaws.com/tejas-dev/reports/sales.csv

Virtual-hosted-style URL (current standard):

https://<bucket-name>.s3.<region>.amazonaws.com/<object-key>
https://tejas-dev.s3.ap-south-1.amazonaws.com/reports/sales.csv

Breaking it down:

https://tejas-dev.s3.ap-south-1.amazonaws.com/reports/2026/sales.csv
        │          │   │                        │
        bucket     │   region                   object key (full path)
                   S3 service

URL Validity

An S3 object URL is only valid while:

The object actually exists in the bucket
The object is publicly accessible (or you're using a pre-signed URL with a valid expiry)

If you delete the object, the URL returns a 404 — the URL structure stays the same but there's nothing at that address.

Pre-Signed URLs

For private objects, you can generate a pre-signed URL — a temporary link with an expiry time that grants access without making the object public.

import boto3
s3 = boto3.client('s3')

# Generate a URL valid for 1 hour
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-private-bucket', 'Key': 'secret-file.pdf'},
    ExpiresIn=3600  # seconds
)
print(url)
# https://my-private-bucket.s3.ap-south-1.amazonaws.com/secret-file.pdf?X-Amz-Signature=...&X-Amz-Expires=3600

🎯 Common use case: Share a download link for a private file with a customer for 24 hours — pre-signed URL with ExpiresIn=86400. Link expires automatically.

S3 Storage Classes

S3 isn't one-size-fits-all for storage. Different classes trade cost vs access speed vs retrieval time.

Storage Class	Use Case	Retrieval	Cost
S3 Standard	Frequently accessed data	Milliseconds	Highest
S3 Intelligent-Tiering	Unknown or changing access patterns	Milliseconds	Moderate (auto-tiered)
S3 Standard-IA	Infrequent access, needs fast retrieval	Milliseconds	Lower storage, retrieval fee
S3 One Zone-IA	Infrequent, reproducible data, one AZ only	Milliseconds	Lower than Standard-IA
S3 Glacier Instant	Archive, accessed once a quarter	Milliseconds	Very low
S3 Glacier Flexible	Archive, minutes-to-hours retrieval ok	Minutes to hours	Very low
S3 Glacier Deep Archive	Long-term archive, accessed once a year	Up to 12 hours	Cheapest

Simple decision guide:

Serving to users daily?              → Standard
Don't know access pattern?           → Intelligent-Tiering (auto-manages)
Accessed once a month, need fast?    → Standard-IA
Old logs, rarely accessed?           → Glacier Instant Retrieval
Legal archive, won't need for years? → Glacier Deep Archive

Lifecycle Policies (Auto-Tiering)

Just like EFS, you can set rules to automatically move objects between storage classes:

Day 0    → Upload to Standard
Day 30   → Auto-move to Standard-IA
Day 90   → Auto-move to Glacier Instant
Day 365  → Auto-move to Glacier Deep Archive
Day 2555 → Auto-delete (7 years, for compliance)

🎯 Cert tip: "How do you minimize S3 storage costs for logs that need to be kept for 7 years but rarely accessed after 30 days?" → Lifecycle policy transitioning to Glacier.

S3 Versioning

When versioning is enabled, S3 keeps every version of every object — overwrites create a new version instead of replacing, and deletes add a "delete marker" instead of actually deleting.

Upload file.txt (v1)      → Version ID: aaa111
Upload file.txt (v2)      → Version ID: bbb222  ← current version
Upload file.txt (v3)      → Version ID: ccc333  ← current version

GET file.txt              → returns v3 (current)
GET file.txt?versionId=aaa111 → returns v1
DELETE file.txt           → adds delete marker, v3 still exists

Why versioning matters:

Accidental overwrite? Restore previous version instantly
Accidental delete? Remove the delete marker to recover
Required for Cross-Region Replication
Required for S3 Object Lock (compliance)

⚠️ Cost warning: Versioning stores every version of every file. An object updated 100 times = 100 stored copies. Enable Lifecycle Policies to expire old versions, otherwise storage costs grow silently.

S3 Encryption — Four Methods

Encryption in S3 has two dimensions: at rest (stored on disk) and in transit (moving over the network). Transit is always HTTPS — the four methods below are all about at rest.

SSE-S3 (Server-Side Encryption with S3-Managed Keys)

AWS manages everything — keys are created, rotated, and used by S3 automatically
Enabled by default on all new buckets (since Jan 2023)
Zero configuration needed
AES-256 encryption
You have no control over the keys

Upload file → S3 auto-encrypts with its own key → stored encrypted
Download → S3 auto-decrypts → you get the file
You never see or touch the key

Use when: You need encryption but don't have compliance requirements about key control.

SSE-KMS (Server-Side Encryption with AWS KMS Keys)

Used mostly

Upload flow:

You upload file.pdf
      ↓
S3 sends request to AWS KMS: "Give me a data encryption key"
      ↓
KMS generates a data key (encrypted + plaintext versions)
      ↓
S3 uses plaintext key to encrypt your file
      ↓
S3 stores: encrypted file + encrypted key (plaintext key discarded)
      ↓
Your file sits on disk: fully encrypted ✅

Download flow:

You request file.pdf
      ↓
S3 checks your IAM permissions (must have s3:GetObject)
      ↓
S3 checks your KMS permissions (must have kms:Decrypt)
      ↓
S3 sends encrypted key to KMS: "Decrypt this"
      ↓
KMS decrypts the key → returns plaintext key
      ↓
S3 uses plaintext key to decrypt your file
      ↓
You receive the original file ✅

Why SSE-KMS over SSE-S3?

You control the KMS key (create, rotate, disable, delete)
Full audit trail — every KMS key usage logged in CloudTrail
Granular access control — you can restrict who can use the key via KMS key policy
Required by many compliance standards (HIPAA, PCI-DSS, FedRAMP)

Downside: Every object upload/download = one KMS API call. KMS has a cost per API call and rate limits.

SSE-C (Server-Side Encryption with Customer-Provided Keys)

You provide your own encryption key with each request
AWS uses it to encrypt/decrypt, but never stores your key
You are 100% responsible for key management

Upload: You send (file + your-key) → S3 encrypts → discards your-key
Download: You send (object-key + your-key) → S3 decrypts → returns file
Lose your key = lose your data permanently (no recovery)

Use when: You have strict regulatory requirements that AWS must never hold your encryption key even temporarily.

CSE (Client-Side Encryption)

You encrypt the file before uploading it to S3
S3 just stores whatever bytes you send — it has no idea the content is encrypted
You manage all encryption/decryption on your side

Your app: file → encrypt locally → send encrypted bytes → S3 stores them
Your app: download encrypted bytes → decrypt locally → original file
AWS sees only ciphertext — never the original data

Use when: Maximum data confidentiality — even AWS employees cannot read your data.

Encryption Comparison

Method	Who manages key	AWS sees plaintext?	Audit trail	Use when
SSE-S3	AWS (automatic)	Internally yes	Basic	Default, no compliance needs
SSE-KMS	You (via KMS)	Internally yes	Full (CloudTrail)	Compliance, audit requirements
SSE-C	You	Never	None	Strict key control
CSE	You	Never	None	Maximum confidentiality

S3 Bucket Keys — Reducing KMS Costs by 99%

With SSE-KMS, every single object upload/download triggers a KMS API call. For buckets with millions of objects, this generates millions of KMS calls — which have both a per-call cost and API rate limits.

S3 Bucket Keys solve this:

Without Bucket Key:

100 object uploads → 100 KMS API calls → $$$

With Bucket Key enabled:

S3 generates one bucket-level data key from KMS
That key is cached in S3 for a time period
100 object uploads → 1 KMS API call + 99 local operations → $ (99% fewer calls)

Analogy: Instead of going to the bank (KMS) to get cash for every purchase, you withdraw a stack of cash once and use it locally for many purchases.

Reduces KMS API costs by up to 99%
Reduces KMS request rate (avoids throttling)
Transparent — encryption strength is unchanged
Enable it when creating the bucket or in bucket properties

🎯 Cert tip: "A company uses SSE-KMS on an S3 bucket with millions of daily uploads and is hitting KMS rate limits. What's the fix?" → Enable S3 Bucket Keys.

AWS CloudTrail + S3 Audit Logging

CloudTrail records every API call made to AWS services — who did what, when, from where. For S3 with KMS encryption, it becomes your full audit trail.

What CloudTrail logs for S3 + KMS:

Who:    IAM user/role that accessed the object
What:   s3:GetObject, s3:PutObject, kms:Decrypt, kms:GenerateDataKey
When:   Timestamp of the request
Where:  Source IP address
Result: Success or failure (and why if failed)

Two types of S3 events in CloudTrail:

Management Events — bucket-level operations (CreateBucket, DeleteBucket, PutBucketPolicy) — logged by default
Data Events — object-level operations (GetObject, PutObject, DeleteObject) — must be explicitly enabled (generates high volume)

S3 Server Access Logging (separate from CloudTrail):

Detailed request logs written directly to another S3 bucket
Every HTTP request to the bucket: requester IP, request type, response code, bytes transferred
Good for access pattern analysis and billing investigation

🎯 DevOps/Security use case: Enable CloudTrail data events for your S3 bucket to detect unauthorized access. If someone outside your organization downloads a sensitive file, CloudTrail has the full record — user, IP, timestamp, object key.

Bucket Policies — Object Ownership & ACL Settings

When multiple AWS accounts are involved (or when files are uploaded by external parties), there are three ownership models that control who owns uploaded objects and how ACLs work.

Three Object Ownership Settings

1. Bucket Owner Enforced (Recommended)

All objects in the bucket are always owned by the bucket owner
ACLs are completely disabled
All access must be controlled via bucket policies or IAM policies only
Simplest and most secure model

External account uploads file → bucket owner automatically owns it
No ACL confusion, no cross-account ownership issues

2. Bucket Owner Preferred

Bucket owner takes ownership IF the uploader includes bucket-owner-full-control ACL
ACLs are still enabled
Gives flexibility while nudging toward central ownership

External account uploads with bucket-owner-full-control ACL
→ bucket owner gets ownership ✅
External account uploads without that ACL
→ external account retains ownership ⚠️

3. Object Writer

The account that uploads an object owns it (legacy default behavior)
Can cause situations where bucket owner cannot read/delete objects in their own bucket
ACLs enabled

✅ Best practice: Always use Bucket Owner Enforced for new buckets. It's simpler, removes ACL complexity, and avoids cross-account ownership confusion that has caused real security incidents.

S3 Access Control — Full Picture

S3 has multiple layers of access control that work together:

Request to access S3 object
        │
        ▼
1. IAM Policy — does the requester's IAM user/role allow this?
        │
        ▼
2. Bucket Policy — does the bucket policy allow this request?
        │
        ▼
3. S3 Block Public Access — is public access blocked at account/bucket level?
        │
        ▼
4. ACL (if enabled) — does the object/bucket ACL permit access?
        │
        ▼
Access Granted or Denied

All applicable policies are evaluated together. An explicit Deny at ANY layer = denied, regardless of allows elsewhere.

Bucket Policy

A JSON document attached to the bucket that defines who can access it and what they can do:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowEC2Access",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:role/EC2-S3-Role"
      },
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyDeleteForEveryone",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

S3 Block Public Access (4 Settings)

AWS added this specifically to prevent accidental data exposure — one of the most common cloud security incidents.

Setting	What it blocks
Block public ACLs	Rejects PUT requests with public ACLs
Ignore public ACLs	Ignores existing public ACLs
Block public bucket policies	Rejects bucket policies that grant public access
Restrict public bucket policies	Restricts public access even if bucket policy allows it

✅ Best practice: Enable all 4 Block Public Access settings at the account level. Explicitly disable only for specific buckets that genuinely need public access (e.g., static website hosting).

Explicit Deny — The Override Rule

This is one of the most important concepts in all of AWS security — and it applies everywhere, not just S3.

The rule:

Explicit Deny ALWAYS wins over any Allow, from any source

Policy evaluation order:

1. Is there an Explicit Deny anywhere? → DENIED (done, nothing else checked)
2. Is there an Explicit Allow? → ALLOWED
3. Neither? → DENIED (implicit deny is the default)

Why this matters in practice:

Scenario: User has AdministratorAccess (allows everything)
          SCP says: Deny s3:DeleteBucket

Result: User CANNOT delete S3 buckets — despite having AdministratorAccess
The Deny wins. Always.

Real example in a bucket policy:

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": "arn:aws:s3:::critical-data/*",
  "Condition": {
    "Bool": {
      "aws:SecureTransport": "false"
    }
  }
}

This denies all S3 actions if the request isn't using HTTPS — even if the user has full S3 permissions elsewhere. The Deny wins.

🎯 Interview tip: "A user has S3FullAccess but can't delete objects from a specific bucket. Why?" → There's likely an explicit Deny somewhere — in a bucket policy, SCP, or permission boundary. Explicit Deny always overrides explicit Allow.

VPC — Brief Introduction

VPC = Virtual Private Cloud — your own logically isolated private network inside AWS.

Think of it as buying a plot of land inside a large city (AWS). Within your plot, you decide the layout — which areas are public-facing, which are private, what gets connected to the internet.

Why it's relevant to S3 here: By default, S3 is accessed over the public internet (via HTTPS). For security, you can set up a VPC Endpoint for S3 — a private connection from your VPC directly to S3 without traffic leaving the AWS network.

Without VPC Endpoint:
EC2 → public internet → S3 (traffic exits AWS network)

With VPC Endpoint (Gateway type):
EC2 → VPC Endpoint → S3 (traffic stays inside AWS backbone)
     No internet gateway needed
     More secure, no data transfer costs for S3

VPC will be covered in depth in a dedicated session. For now: VPC = your private network, and S3 can be accessed privately through it via Gateway Endpoints.

S3 CLI Commands

# List all buckets
aws s3 ls

# List objects in a bucket
aws s3 ls s3://my-bucket/
aws s3 ls s3://my-bucket/reports/ --recursive

# Create a bucket
aws s3 mb s3://my-new-bucket-name --region ap-south-1

# Upload a file
aws s3 cp local-file.txt s3://my-bucket/uploads/file.txt

# Upload entire folder
aws s3 cp ./my-folder s3://my-bucket/my-folder/ --recursive

# Download a file
aws s3 cp s3://my-bucket/file.txt ./local-file.txt

# Sync local folder to S3 (only uploads new/changed files)
aws s3 sync ./local-folder s3://my-bucket/folder/

# Delete an object
aws s3 rm s3://my-bucket/old-file.txt

# Delete all objects in a folder
aws s3 rm s3://my-bucket/old-folder/ --recursive

# Delete a bucket (must be empty first)
aws s3 rb s3://my-bucket

# Delete bucket and all contents
aws s3 rb s3://my-bucket --force

# Copy between buckets (cross-region)
aws s3 cp s3://source-bucket/file.txt s3://dest-bucket/file.txt \
  --source-region ap-south-1 --region us-east-1

# Set storage class on upload
aws s3 cp file.txt s3://my-bucket/ --storage-class STANDARD_IA

# Generate pre-signed URL (valid 1 hour)
aws s3 presign s3://my-bucket/private-file.pdf --expires-in 3600

# Put bucket policy
aws s3api put-bucket-policy \
  --bucket my-bucket \
  --policy file://policy.json

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled

boto3 — S3 with Python

import boto3

s3 = boto3.client('s3', region_name='ap-south-1')
s3_resource = boto3.resource('s3')

# ── List buckets ────────────────────────────────────────────
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(f"Bucket: {bucket['Name']} | Created: {bucket['CreationDate']}")

# ── Create bucket ───────────────────────────────────────────
s3.create_bucket(
    Bucket='my-new-bucket-2026',
    CreateBucketConfiguration={'LocationConstraint': 'ap-south-1'}
)

# ── Upload file ─────────────────────────────────────────────
s3.upload_file('local-file.txt', 'my-bucket', 'uploads/file.txt')

# ── Upload with SSE-KMS encryption ─────────────────────────
s3.upload_file(
    'sensitive.pdf',
    'my-bucket',
    'secure/sensitive.pdf',
    ExtraArgs={
        'ServerSideEncryption': 'aws:kms',
        'SSEKMSKeyId': 'arn:aws:kms:ap-south-1:123456789:key/abc-123'
    }
)

# ── Download file ───────────────────────────────────────────
s3.download_file('my-bucket', 'uploads/file.txt', 'downloaded-file.txt')

# ── List objects in bucket ──────────────────────────────────
response = s3.list_objects_v2(Bucket='my-bucket', Prefix='reports/')
for obj in response.get('Contents', []):
    print(f"Key: {obj['Key']} | Size: {obj['Size']} bytes")

# ── Delete object ───────────────────────────────────────────
s3.delete_object(Bucket='my-bucket', Key='old-file.txt')

# ── Generate pre-signed URL ─────────────────────────────────
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'private-doc.pdf'},
    ExpiresIn=3600
)
print(f"Share this URL (valid 1 hour): {url}")

# ── Apply bucket policy ─────────────────────────────────────
import json
policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
        ],
        "Condition": {"Bool": {"aws:SecureTransport": "false"}}
    }]
}
s3.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

⚡ Quick Revision

JSON vs YAML
  JSON = braces/quotes/commas, API responses, IAM policies
  YAML = indentation/colons, CloudFormation, SAM
  Both used for infrastructure-as-code — know both

API CALLS
  GET    = read/retrieve
  PUT    = create/replace
  POST   = create/trigger
  DELETE = remove
  HEAD   = metadata only

S3 OBJECT URL
  https://<bucket>.s3.<region>.amazonaws.com/<key>
  Valid only while object exists and access is permitted
  Pre-signed URL = temporary access for private objects

STORAGE CLASSES (cost vs speed)
  Standard          → frequent access, fastest, most expensive
  Intelligent-Tiering → unknown patterns, auto-optimizes
  Standard-IA       → monthly access, fast retrieval fee
  Glacier Instant   → quarterly access, instant retrieval
  Glacier Deep      → annual access, 12hr retrieval, cheapest

ENCRYPTION
  SSE-S3  → AWS manages key, default, zero config
  SSE-KMS → you control key via KMS, audit trail, compliance
  SSE-C   → you provide key per request, AWS never stores it
  CSE     → you encrypt before upload, AWS sees ciphertext only

S3 BUCKET KEYS
  Reduces KMS API calls by 99%
  One bucket-level key cached, used for multiple objects
  Enable for high-volume SSE-KMS buckets

BUCKET OWNERSHIP
  Owner Enforced  = ACLs off, bucket owner owns all (use this)
  Owner Preferred = owner gets ownership if ACL included
  Object Writer   = uploader owns (legacy, avoid)

EXPLICIT DENY
  Always overrides any Allow, from any source
  Evaluation: Explicit Deny → Explicit Allow → Implicit Deny

CLOUDTRAIL
  Logs every API call: who, what, when, where, result
  Data events (object-level) must be explicitly enabled
  Use for security auditing of S3 + KMS access

💼 Interview Questions

Q1: What is the difference between an S3 bucket and an S3 object?
A bucket is the container — like a top-level folder — created in a specific Region with a globally unique name. An object is the actual file stored inside the bucket, uniquely identified by its key (path). The bucket holds the object; the object is the data.

Q2: What makes an S3 object URL unique and when does it stop working?
The URL is formed from the bucket name, region, and object key — all of which are unique. It stops working when the object is deleted or when access permissions are revoked. For private objects, you need a pre-signed URL with an expiry time to share temporary access.

Q3: What is SSE-KMS and how does it work?
SSE-KMS uses AWS Key Management Service for encryption. On upload, S3 requests a data key from KMS, encrypts the file with the plaintext key, and stores both the encrypted file and the encrypted key (discarding the plaintext key). On download, S3 verifies IAM and KMS permissions, asks KMS to decrypt the encrypted key, then decrypts and returns the file. You control the KMS key and every usage is logged in CloudTrail.

Q4: What are S3 Bucket Keys and why would you enable them?
S3 Bucket Keys are a bucket-level data key cached in S3, used to encrypt multiple objects instead of calling KMS for every single operation. This reduces KMS API calls by up to 99%, significantly reducing cost and avoiding KMS rate limit throttling — essential for high-throughput S3 buckets using SSE-KMS.

Q5: What is the difference between SSE-KMS and SSE-C?
SSE-KMS means AWS generates and manages the data keys inside KMS — you control the master key but AWS handles the actual encryption. SSE-C means you provide your own encryption key with every request — AWS uses it and immediately discards it, never storing it. If you lose your SSE-C key, your data is gone forever.

Q6: An IAM user has S3FullAccess but cannot delete objects from a bucket. Why?
There's an explicit Deny somewhere overriding the Allow. It could be in the bucket policy, an SCP at the Organizations level, or a permission boundary. Explicit Deny always wins over any Allow, from any source.

Q7: What is Bucket Owner Enforced and why is it recommended?
It disables ACLs entirely and ensures the bucket owner always owns all objects regardless of who uploaded them. It's the simplest and most secure model — no ACL confusion, no cross-account ownership issues, and access is controlled purely through bucket policies and IAM.

Q8: Write a bucket policy that denies all HTTP (non-HTTPS) access.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::my-bucket",
      "arn:aws:s3:::my-bucket/*"
    ],
    "Condition": {
      "Bool": {"aws:SecureTransport": "false"}
    }
  }]
}

🔬 Practice Tasks

Create and configure a bucket: Create an S3 bucket with versioning enabled, Block Public Access on, and a lifecycle policy that moves objects to Standard-IA after 30 days and Glacier after 90 days.
Write bucket policies: Write a JSON bucket policy that: (a) allows your IAM user to read all objects, (b) allows an EC2 role to upload objects, and (c) denies all non-HTTPS requests.
SSE-KMS lab: Enable SSE-KMS on a bucket. Upload a file. Download it. Open CloudTrail and find the kms:GenerateDataKey and kms:Decrypt events corresponding to your upload and download.
Pre-signed URL: Using boto3, upload a private file to S3, generate a pre-signed URL valid for 5 minutes, open it in your browser, wait 5 minutes, try again — confirm it expires.
CLI practice: Using AWS CLI, sync a local folder to S3, list the objects, update one file locally, sync again, and confirm only the changed file was re-uploaded (not everything).
Versioning test: Enable versioning on a bucket. Upload a file, modify it, upload again with the same key, upload a third time. List all versions. Retrieve v1. Delete the latest version and confirm v2 becomes current.
Write the JSON and YAML: Without looking at notes, write from scratch: (a) a CloudFormation YAML snippet that creates an S3 bucket with versioning, (b) an S3 bucket policy in JSON that makes all objects in /public/* readable by everyone.

AWS Session 7 — S3 Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS ELB & Auto Scaling — Building Self-Healing, Scalable Infrastructure

Tejas Shinkar — Fri, 03 Jul 2026 01:51:46 +0000

Elastic Load Balancing & Auto Scaling

Traffic Distribution · ASG · Scaling Policies · ALB vs NLB · Health Checks · Zero-Downtime Deployments

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session covers the two services that make AWS infrastructure truly production-grade: ELB and Auto Scaling.

📋 Topics Covered

#	Topic	Type
1	Purpose of Load Balancing	Concept
2	How Load Balancing Works (Algorithms)	Concept + Interview
3	Horizontal vs Vertical Scaling	Concept + Interview
4	Scaling Policies — Dynamic, Scheduled, Predictive	Concept + Cert
5	Auto Scaling Group (ASG) — Core Config	Concept + Lab
6	ASG Lifecycle & Key Metrics	Concept + Cert
7	Health Checks & Replacement Policies	Concept + Lab
8	Session Stickiness	Concept + Interview
9	Connection Draining (Deregistration Delay)	Concept + Interview
10	Instance Refresh — Zero-Downtime Deployments	Concept + DevOps
11	Load Balancer Types — ALB vs NLB vs GWLB	Concept + Cert
12	Target Groups & Health Check Config	Concept + Lab
13	ALB Deep Dive — Routing Rules	Concept + Cert
14	Architecture — ALB + ASG	Concept + DevOps
15	Monitoring & Troubleshooting	Concept + DevOps
16	Best Practices	DevOps
17	AWS CLI Commands	Practical
18	Lab — Create ELB + Auto Scaling Group (7 Steps)	Lab
19	Cleanup Checklist	Lab
20	Assignment	Practice

Why Load Balancing Exists

Imagine a popular restaurant with one cashier — as soon as the lunch rush hits, the queue grows, the cashier gets overwhelmed, and eventually everything grinds to a halt. The fix? More cashiers, with a manager at the door directing customers to whoever is free.

A Load Balancer is that manager.

It sits in front of your EC2 instances and distributes incoming traffic across them — so no single server gets overwhelmed, and if one fails, the others keep running.

Without a Load Balancer:

Single point of failure — one server down means your app is down
No horizontal scaling — you're stuck with one server
No health checking — dead servers still receive traffic
Manual failover — someone has to intervene

With a Load Balancer:

High availability across multiple AZs
Horizontal scalability — add more servers on demand
Automatic health checks — dead servers removed instantly
Automatic failover — zero manual intervention
SSL/TLS termination — offloads HTTPS decryption from backend servers

How Load Balancing Works

Request flow — what actually happens:

1. User's browser sends HTTP request to your domain
2. DNS resolves your domain → returns ALB's IP address
3. Request hits the Load Balancer
4. LB picks a healthy backend EC2 instance
5. Forwards request to that instance
6. Instance processes and responds
7. LB returns the response to the user

Load Distribution Algorithms:

Algorithm	How it works	AWS Usage
Round Robin	Requests go in rotation: 1→2→3→1→2→3	ALB default
Least Connections	Route to instance with fewest active connections	Good for variable request lengths
IP Hash	Client IP always maps to same backend	Sticky sessions
Weighted	Instance A gets 70%, B gets 30%	Canary deployments
Flow Hash	Hash of protocol+IP+port	NLB default for TCP/UDP

🎯 Cert Tip: ALB uses weighted round robin by default. NLB uses flow hash for TCP/UDP traffic.

Horizontal vs Vertical Scaling

This is one of the most fundamental concepts in cloud architecture — and a guaranteed interview question.

Analogy: Your app is a delivery service. Vertical scaling = buying a bigger truck. Horizontal scaling = buying more trucks.

Vertical Scaling (Scale Up)

Upgrade the existing instance — stop it, change the type, restart.

t3.micro → t3.medium → t3.large → t3.xlarge

✅ Simple — no distributed system complexity
✅ Better for databases (data consistency is easier)
❌ Has a hard ceiling — biggest instance type is the limit
❌ Requires downtime during the type change
❌ Cannot auto-scale

Horizontal Scaling (Scale Out)

Add more instances — distribute load across them.

1 × t3.micro → 2 × t3.micro → 4 × t3.micro → back to 2

✅ Theoretically unlimited scalability
✅ Fault tolerant — one fails, others keep running
✅ Scales dynamically with Auto Scaling Groups
✅ No downtime when scaling
❌ Requires a Load Balancer to distribute traffic
❌ Requires stateless app design (or sticky sessions for state)

🎯 Cloud-native best practice: always use horizontal scaling with ASG. Vertical scaling is for legacy systems or databases that can't easily distribute state.

Scaling Policies — Three Types

How does AWS know when to scale? Scaling Policies define the trigger.

1. Dynamic Scaling — Target Tracking (Most Common)

Monitor a metric and automatically adjust to keep it at a target value.

Metric: Average CPU Utilization
Target: 50%

CPU rises to 75% → ASG adds instances until CPU ≈ 50%
CPU drops to 20% → ASG removes instances until CPU ≈ 50%

AWS creates two CloudWatch alarms automatically — one for scale-out, one for scale-in. You don't configure alarms manually.

Best for: Most production workloads. Simplest to set up and maintain.

2. Scheduled Scaling

Scale at pre-defined times — for predictable, known traffic patterns.

Every weekday 8:30 AM  → set Desired = 6  (office hours ramp up)
Every weekday 7:00 PM  → set Desired = 2  (evening wind down)
Every Saturday 12:00 AM → set Desired = 8 (weekend traffic spike)

Best for: Business-hours traffic, batch jobs at fixed times, known seasonal spikes.

3. Predictive Scaling (ML-based)

AWS analyses at least 2 weeks of historical metric data, forecasts future demand, and scales EC2 instances before traffic arrives — not in reaction to it.

Historical pattern: Every Friday at 5PM, traffic doubles
Predictive: ASG pre-scales at 4:45PM — instances ready before surge hits

Best for: Recurring, predictable patterns where reactive scaling causes latency spikes.

🎯 Cert Tip: Predictive Scaling requires a minimum of 2 weeks of historical data to build its forecast model.

Auto Scaling Group (ASG)

An ASG is the component that actually manages your EC2 fleet — launching, terminating, and replacing instances automatically.

Analogy: ASG is like a staffing agency for your servers. You tell it: "I need at least 2 people, ideally 4, maximum 10 — and if anyone quits or gets sick, hire a replacement immediately."

The Three Capacity Numbers (Core Concept)

MAX     [4] ──── Hard ceiling. ASG will NEVER exceed this, even under extreme load
DESIRED [2] ──── ASG always tries to maintain exactly this many healthy instances
MIN     [1] ──── Hard floor. ASG will NEVER go below this, even at zero traffic

ASG constantly compares running instances against Desired capacity. If an instance fails a health check, it terminates it and launches a replacement — no human intervention needed.

ASG Configuration Components

Setting	What it does
Launch Template	Blueprint for every EC2 ASG launches — AMI, instance type, SG, User Data
Min / Desired / Max	Capacity boundaries
Availability Zones	Which AZs to spread instances across (always 2+)
Load Balancer / Target Group	Where to register launched instances
Health Check Type	EC2 (VM-level) or ELB (app-level) — always use ELB
Scaling Policies	Dynamic, scheduled, or predictive rules
Termination Policy	Which instances to remove during scale-in (default: oldest)

✅ Always use Launch Templates not Launch Configurations — templates support versioning, Instance Refresh, and newer EC2 features.

ASG Lifecycle — Instance States

Scale-out triggered
      ↓
  Pending ──── being launched by ASG
      ↓
  InService ── running, healthy, serving traffic ← normal operating state
      ↓
  Draining ─── marked for removal, existing connections completing
      ↓
  Terminated ─ deleted, no longer in ASG

  Unhealthy ── failed health check → immediately replaced with new instance

Health Checks & Replacement Policies

Two Types of Health Checks

EC2 Health Check (VM level)
Checks if the instance is running — system and instance status checks pass. Doesn't know if your application is actually responding.

ELB Health Check (App level)
The ALB sends a real HTTP request to your configured path (e.g., /health). If the app doesn't return HTTP 200, the instance is marked unhealthy.

✅ Always use ELB health checks for web applications. An instance can be "running" while Apache is crashed inside — EC2 checks miss this entirely. ELB checks catch it within 60 seconds.

Health Check Configuration

Setting	Default	What it means
Interval	30 seconds	How often checks are sent
Timeout	5 seconds	How long to wait for response
Healthy Threshold	2 consecutive successes	Instance declared healthy
Unhealthy Threshold	2 consecutive failures	Instance declared unhealthy
Grace Period	300 seconds	Time after launch before checks begin (allows app boot time)

Self-Healing Flow

Health check fails twice
      ↓
ALB marks instance "Unhealthy" → stops sending traffic to it
      ↓
ASG receives notification
      ↓
ASG terminates the unhealthy instance
      ↓
ASG launches a new replacement from Launch Template
      ↓
Replacement passes health checks → added to Target Group
      ↓
Total time: ~5-10 minutes, fully automated ✅

Session Stickiness (Sticky Sessions)

By default, ALB routes each request to any healthy instance via round robin — a user's second request might go to a different server than their first. For stateless apps that's fine. For stateful apps (shopping carts, auth sessions), the session exists only on one server.

Sticky Sessions fixes this: ALB sets a cookie in the browser that pins the user to one specific backend instance for the session duration.

User logs in → ALB routes to EC2-1 → sets cookie: "stick to EC2-1"
Next request → user sends cookie → ALB routes to EC2-1 again ✅
Session expires / cookie deleted → back to normal round robin

When to use: Shopping carts, user authentication, stateful apps with in-memory sessions.

Downsides:

Uneven load distribution
If the pinned instance fails, the session is lost anyway

🎯 Proper long-term fix: Externalize session state to ElastiCache (Redis) or DynamoDB so any instance can serve any user without needing stickiness.

Connection Draining (Deregistration Delay)

When an instance is removed (scale-in, replacement, deployment), what happens to users mid-request on that instance?

Without draining: requests are immediately dropped → users get errors.

Connection Draining:

Instance marked for removal
      ↓
ALB immediately stops sending NEW requests to this instance
      ↓
ALB waits for EXISTING in-flight requests to finish
      ↓
Default wait: up to 300 seconds
      ↓
After all connections finish (or timeout) → instance terminated
      ↓
Zero dropped requests ✅

This is what enables zero-downtime scaling and deployments.

Instance Refresh — Zero-Downtime Deployments

When you update your application (new AMI, new config), you need to replace all running instances with the new version. Instance Refresh automates this within ASG.

Refresh strategies:

Strategy	Method	Speed	Risk
Rolling	One instance at a time	Slowest	Lowest
Rolling with Stagger	X% in parallel (25%→50%→75%)	Medium	Medium
Blue/Green	All new instances launched, traffic switched, old terminated	Fastest	Higher cost

Workflow:

Update Launch Template (new AMI version)
      ↓
Start Instance Refresh in ASG console
      ↓
ASG launches new instance from updated template
      ↓
Health checks pass on new instance
      ↓
Old instance: connection draining → terminated
      ↓
Repeat for all instances
      ↓
All instances running new AMI — zero downtime ✅

🎯 DevOps relevance: Instance Refresh is the ASG equivalent of a rolling update in Kubernetes. Same concept, different tool.

Load Balancer Types — ALB vs NLB vs GWLB

	ALB	NLB	GWLB
OSI Layer	Layer 7 (Application)	Layer 4 (Transport)	Layer 3 (Network)
Protocol	HTTP, HTTPS	TCP, UDP, TLS	All IP protocols
Routing basis	Path, host, header, query string	IP + Port (flow hash)	Transparent proxy
Latency	~25-100ms	<100 microseconds	Very low
Static IP	❌ No	✅ Yes	✅ Yes
SSL Termination	✅ Yes	✅ Yes	❌ No
Use Case	Web apps, APIs, microservices	Gaming, IoT, VoIP, real-time	Firewalls, IDS/IPS appliances

Decision guide:

Web app or REST API?                 → ALB (almost always)
Ultra-low latency TCP/UDP required?  → NLB
Traffic must pass through a firewall/security appliance? → GWLB

⚠️ Classic Load Balancer (CLB) exists but is legacy — never use for new deployments.

🎯 Cert scenarios: "Which LB supports path-based routing?" → ALB only. "Which provides a static IP?" → NLB. "Which for real-time gaming at massive scale?" → NLB.

ALB Deep Dive — Routing Rules

ALB operates at Layer 7 — it understands HTTP. This enables routing based on the actual content of each request.

Listener Rules (evaluated top-to-bottom):

Rule 1: Host = api.example.com
        → Forward to: api-target-group (3x t3.medium)

Rule 2: Host = admin.example.com
        → Forward to: admin-target-group (restricted access)

Rule 3: Path = /images/*
        → Forward to: media-target-group (or S3)

Rule 4: Path = /api/*
        → Forward to: api-target-group

Rule 5: HTTP (port 80) — any path
        → Redirect: 301 to HTTPS

Default Rule: no other rule matched
        → Forward to: web-target-group (main homepage)

One ALB can route to multiple backend services — this is the foundation of microservices architecture on AWS.

Target Groups & Health Check Config

A Target Group is a named pool of EC2 instances (or IPs, or Lambdas) that the ALB routes traffic to. One ALB → multiple target groups via listener rules.

Setting	Value
Target Type	Instances
Protocol	HTTP or HTTPS
Port	App's listening port (80, 8080, 3000, etc.)
VPC	Must match your EC2 instances
Health Check Path	`/health` — must return HTTP 200-299

⚠️ When using ASG — never manually register instances to the Target Group. Let ASG handle registration/deregistration automatically as instances launch and terminate.

Architecture — ALB + ASG (Production Standard)

                     Internet Users
                           │
                       Route 53
                  (domain → ALB DNS)
                           │
               Application Load Balancer
             (Internet-facing, 2+ AZs, SG-ALB)
                           │
                     Target Group
               (health-checked instance pool)
                           │
            Auto Scaling Group (min=1, desired=2, max=4)
         ┌─────────────────┼──────────────────┐
         │                 │                  │
    EC2 (AZ-a)         EC2 (AZ-b)        EC2 (AZ-b)
    SG-EC2, :80        SG-EC2, :80       SG-EC2, :80
         │
     (shared state)
    RDS Database
    or ElastiCache

CloudWatch CPU metric → Scaling Policy → ASG scales in/out

Three architectural pillars achieved:

High Availability — multi-AZ ALB and instances
Fault Tolerance — ASG self-heals on any instance failure
Cost Efficiency — scale down at night, up during peaks

Monitoring & Troubleshooting

Key CloudWatch Metrics

ALB Metrics:

Metric	Alert When
`TargetResponseTime`	> 500ms sustained
`RequestCount`	Sudden unexpected drop
`HTTPCode_Target_5XX`	> 0 sustained
`HealthyHostCount`	Below Desired capacity
`UnHealthyHostCount`	Greater than 0

ASG Metrics:

Metric	What it shows
`GroupDesiredCapacity`	Target count
`GroupInServiceInstances`	Healthy, running count
`GroupPendingInstances`	Currently launching
`GroupTerminatingInstances`	Being removed

Common Issues & Fixes

UnHealthyHostCount > 0

Check: SG-EC2 allows port 80 FROM SG-ALB (not from internet)
Check: Health check path actually returns 200 (try curl from EC2)
Check: App running → systemctl status httpd
Check: Grace period → did instance have 300s to boot before checks started?

ASG not scaling despite high CPU

Check: Scaling policy threshold — is CPU actually crossing it?
Check: Max capacity — already at the ceiling?
Check: Cooldown — previous scale action still in cooldown?
Check: CloudWatch alarm — is it actually in ALARM state?

Requests timing out after ALB

Check: EC2 instances responding? Try curl from inside the VPC
Check: Security group — SG-EC2 must allow HTTP FROM SG-ALB only
Check: App logs — any errors being thrown?

Session lost when instance replaced

Short-term: Enable sticky sessions on the Target Group
Long-term: Move session state to ElastiCache (Redis) or DynamoDB

Best Practices

ALB
  ✅ Always deploy across 2+ AZs
  ✅ Use path-based routing to consolidate microservices on one ALB
  ✅ Enable access logs for debugging and compliance
  ✅ Add HTTPS listener + redirect HTTP → HTTPS (use ACM for free cert)
  ✅ Use dedicated health check endpoint /health (not just /)

AUTO SCALING
  ✅ Use Target Tracking (simplest) over Step Scaling
  ✅ Set cooldown 300-600 seconds to avoid scaling oscillation
  ✅ Set Grace Period long enough for your app to boot (300s minimum)
  ✅ Use Launch Templates (not Launch Configurations)
  ✅ Test scaling with a load test before going to production
  ✅ Combine Scheduled + Target Tracking for predictable + reactive scaling

HEALTH CHECKS
  ✅ Always use ELB health checks (not EC2 only)
  ✅ Healthy threshold: 2, Unhealthy threshold: 2 (fast detection)
  ✅ Enable connection draining (default 300s)

SECURITY
  ✅ SG-EC2 inbound: HTTP from SG-ALB only (never 0.0.0.0/0)
  ✅ Users hit ALB only — EC2 instances never exposed to internet directly

AWS CLI Commands

# Create Application Load Balancer
aws elbv2 create-load-balancer \
  --name web-app-alb \
  --subnets subnet-aaa subnet-bbb \
  --security-groups sg-alb-xxx \
  --scheme internet-facing \
  --type application

# Create Target Group
aws elbv2 create-target-group \
  --name web-server-tg \
  --protocol HTTP --port 80 \
  --vpc-id vpc-xxx \
  --health-check-path /

# Create Auto Scaling Group
aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name web-app-asg \
  --launch-template LaunchTemplateName=web-server-lt,Version='$Latest' \
  --min-size 1 --max-size 4 --desired-capacity 2 \
  --availability-zones us-east-1a us-east-1b

# Attach Target Group to ASG
aws autoscaling attach-load-balancer-target-groups \
  --auto-scaling-group-name web-app-asg \
  --target-group-arns arn:aws:elasticloadbalancing:...

# Create Target Tracking Policy (CPU 50%)
aws autoscaling put-scaling-policy \
  --auto-scaling-group-name web-app-asg \
  --policy-name cpu-target-tracking \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration \
  '{"TargetValue":50.0,"PredefinedMetricSpecification":{"PredefinedMetricType":"ASGAverageCPUUtilization"}}'

# Describe ASG
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names web-app-asg

# Manually set desired capacity
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name web-app-asg \
  --desired-capacity 3

🧪 LAB — Create ELB + Auto Scaling Group

Security Groups (Create First)

SG-ALB:

Inbound:  HTTP (80) from 0.0.0.0/0
Outbound: All traffic

SG-EC2:

Inbound:  HTTP (80) from SG-ALB  ← reference the SG, not CIDR
Outbound: All traffic

⚠️ SG-EC2's inbound rule must reference SG-ALB as source — not 0.0.0.0/0. This blocks direct internet access to EC2. Only ALB can reach them.

Step 1 — Create Launch Template

EC2 → Launch Templates → Create launch template

Name:            web-server-lt
AMI:             Amazon Linux 2023 (64-bit x86)
Instance Type:   t2.micro / t3.micro (Free Tier)
Key Pair:        your-key-pair
Security Group:  SG-EC2
Storage:         8 GiB gp3

User Data (paste in Advanced Details):

#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/availability-zone)
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/region)

cat > /var/www/html/index.html << EOF
<!DOCTYPE html>
<html>
<body style="font-family:Arial;text-align:center;padding:60px;background:#0A1628;color:#fff;">
  <h1 style="color:#FF9900;">Hello from AWS!</h1>
  <p>Instance ID: <b style="color:#FF9900;">$INSTANCE_ID</b></p>
  <p>Availability Zone: <b style="color:#FF9900;">$AZ</b></p>
  <p>Region: <b style="color:#FF9900;">$REGION</b></p>
</body>
</html>
EOF

✅ Checkpoint: web-server-lt listed in EC2 → Launch Templates

Step 2 — Create Target Group

EC2 → Load Balancing → Target Groups → Create target group

Target type:         Instances
Name:                web-server-tg
Protocol / Port:     HTTP / 80
VPC:                 Default VPC
Health check path:   /
Healthy threshold:   2
Unhealthy threshold: 2
Timeout:             5 seconds
Interval:            30 seconds

⚠️ On "Register targets" page — skip it entirely, click Create directly. ASG will register instances automatically.

✅ Checkpoint: web-server-tg shows status "unused" — this is correct, no instances yet.

Step 3 — Create Application Load Balancer

EC2 → Load Balancers → Create Load Balancer → Application Load Balancer

Name:           web-app-alb
Scheme:         Internet-facing
VPC:            Default VPC
Subnets:        Select 2 subnets in DIFFERENT AZs (mandatory — fails with one)
Security Group: SG-ALB (remove default SG)
Listener:       HTTP : 80 → Forward to web-server-tg

Wait ~2-3 minutes for state to become Active.

✅ Checkpoint: Copy ALB DNS name — you'll need it to test.

Step 4 — Create Auto Scaling Group

EC2 → Auto Scaling Groups → Create Auto Scaling group

Name:             web-app-asg
Launch template:  web-server-lt (Latest)
VPC + Subnets:    Same 2 subnets as ALB

Load balancing:   Attach to existing → web-server-tg | HTTP
Health check:     ELB, Grace period: 300 seconds

Desired: 2  |  Min: 1  |  Max: 4
Scaling: None for now (add in Step 5)

✅ Checkpoint: 2 EC2 instances launch within 1-2 min, both show Healthy in Target Group.

Step 5 — Configure Scaling Policy

ASG → web-app-asg → Automatic scaling → Create dynamic scaling policy

Policy type:   Target tracking scaling
Policy name:   cpu-target-tracking
Metric:        Average CPU Utilization
Target value:  50
Warmup:        300 seconds

AWS auto-creates two CloudWatch alarms (scale-out + scale-in).

Step 6 — Test & Verify

# Open ALB DNS in browser
http://<ALB-DNS-name>
# → "Hello from AWS!" + Instance ID + AZ

# Refresh 5-10 times
# → Instance ID changes = ALB routing to different instances ✅
# → AZ changes = multi-AZ distribution ✅

# Trigger scale-out (SSH into one EC2)
ssh -i your-key.pem ec2-user@<ec2-public-ip>
sudo yum install -y stress
stress --cpu 4 --timeout 600

# Watch in console:
# EC2 → Auto Scaling Groups → Activity tab (new instances launching)
# CloudWatch → Alarms (CPU alarm goes to ALARM state)

Expected timeline after stress starts:

T+0:00  CPU spikes
T+1:00  CloudWatch alarm: OK → ALARM
T+3:00  New EC2 launches
T+5:00  New instance healthy → ALB routes to 3 instances
T+10:00 Stress ends → CPU drops
T+18:00 ASG scale-in → back to Desired=2

Step 7 — Monitor with CloudWatch

CloudWatch → Metrics → EC2 → By Auto Scaling Group
  → web-app-asg → CPUUtilization → Period: 1 min

CloudWatch → Metrics → ApplicationELB
  → Add: RequestCount, HealthyHostCount, TargetResponseTime

🧹 Cleanup — Delete in This Exact Order

1. Auto Scaling Group  → terminates all EC2 instances automatically
   Wait 2-3 min for all instances to fully terminate

2. Application Load Balancer

3. Target Group

4. Launch Template

5. Security Groups  → delete SG-EC2 first, then SG-ALB
   (SG-ALB referenced by SG-EC2 — must delete SG-EC2 first)

6. CloudWatch Alarms  → delete auto-created scaling alarms

💰 ALB = ~$16/month + EC2 costs. Always clean up after labs.

📝 Assignment

Task 1 (Mandatory) — Prove Self-Healing:
Manually terminate one running EC2 instance. Watch ASG Activity tab — confirm it auto-launches a replacement. Verify ALB kept serving traffic with zero downtime.

Task 2 (Intermediate) — Step Scaling Policy:
Create a Step Scaling policy: add 2 instances if CPU > 70%, add 1 if CPU 50-70%, remove 1 if CPU < 20%.

Task 3 (Intermediate) — HTTPS:
Add HTTPS listener (port 443) to ALB. Use AWS Certificate Manager for a free certificate. Add listener rule: HTTP → HTTPS redirect.

Task 4 (Advanced) — Scheduled Scaling:
Scale up to 4 instances at 9AM weekdays, down to 1 at 6PM weekdays. Set times in UTC.

Task 5 (Advanced) — CloudWatch Dashboard:
Build a dashboard with 4 widgets: CPU%, RequestCount, HealthyHostCount, GroupInServiceInstances.

⚡ Quick Revision

LOAD BALANCING
  Manager directing traffic to healthy servers
  Eliminates single point of failure
  ALB algorithm: weighted round robin (default)

SCALING
  Horizontal = more instances (cloud-native, preferred)
  Vertical   = bigger instance (simple, has ceiling, needs downtime)

SCALING POLICIES
  Target Tracking = keep metric at target, AWS manages alarms (use this)
  Scheduled       = scale at specific times
  Predictive      = ML-based, needs 2 weeks history

ASG CAPACITY
  MIN     = hard floor, never below
  DESIRED = ASG always works to maintain this count
  MAX     = hard ceiling, never exceed

HEALTH CHECKS
  EC2  = is instance running? (basic)
  ELB  = is app returning HTTP 200? (always use this)
  Grace Period = time after launch before first check (300s default)

LB TYPES
  ALB  = Layer 7, HTTP/HTTPS, path/host routing → web apps
  NLB  = Layer 4, TCP/UDP, microsecond latency → gaming/IoT
  GWLB = Layer 3, security appliances, firewalls

STICKY SESSIONS
  Cookie pins user to one EC2 for session duration
  Proper fix: external session store (Redis/ElastiCache)

CONNECTION DRAINING
  In-flight requests complete before instance terminates
  Default: 300 seconds | Enables zero-downtime scaling

INSTANCE REFRESH
  Rolling AMI replacement across entire ASG fleet
  Zero-downtime | Kubernetes rolling update equivalent

SECURITY GROUP PATTERN
  SG-ALB: HTTP 80 from 0.0.0.0/0
  SG-EC2: HTTP 80 FROM SG-ALB only
  Direct EC2 access blocked from internet

💼 Interview Questions

Q1: Horizontal vs Vertical scaling — what's the difference and which does AWS recommend?
Horizontal = add more instances, requires LB, unlimited scale, fault tolerant. Vertical = upgrade existing instance, simple, has a hard ceiling, requires downtime. AWS recommends horizontal scaling for cloud-native apps.

Q2: What are Min, Desired, and Max capacity in an ASG?
Min = hard floor, ASG never drops below. Max = hard ceiling, ASG never exceeds. Desired = the target count ASG constantly works to maintain. Scaling policies adjust Desired within Min-Max boundaries.

Q3: Why use ELB health checks instead of EC2 health checks?
EC2 checks only verify the VM is running. ELB checks send a real HTTP request and verify the app returns 200. An EC2 can be "running" while Apache is crashed — EC2 checks miss it, ELB checks catch it within 60 seconds.

Q4: What is Connection Draining?
It allows in-flight requests to complete before an instance is terminated. ALB immediately stops sending new requests but waits up to 300 seconds for active connections to finish — enabling zero-downtime scaling and deployments.

Q5: ALB vs NLB — when do you use each?
ALB for HTTP/HTTPS web apps and microservices — understands application content, supports path/host routing. NLB for ultra-low latency TCP/UDP workloads (gaming, IoT, VoIP) or when a static IP is required.

Q6: Why reference SG-ALB as source in SG-EC2 instead of 0.0.0.0/0?
Referencing SG-ALB as source means only traffic that came through the ALB can reach EC2. Direct internet access to the EC2 IP is blocked. More secure, no IP management needed as instances scale.

Q7: What is Instance Refresh?
ASG's native mechanism to replace all running instances with a new Launch Template version (new AMI or config). It handles draining, replacement, and health checks automatically. Equivalent to a Kubernetes rolling update.

AWS Session 6 — ELB & Auto Scaling | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS EFS Essentials — Shared File Storage Across Multiple EC2 Instances

Tejas Shinkar — Wed, 01 Jul 2026 06:41:40 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session covers Amazon EFS — shared storage that multiple EC2 instances can read/write simultaneously.

📋 Topics Covered

#	Topic	Type
1	What is Amazon EFS	Concept
2	EFS vs EBS vs S3	Concept + Interview
3	EFS Architecture (Mount Targets, AZs, NFS)	Concept + Cert
4	Mounting — What It Means	Concept
5	Mount Targets & Security Group Config	Concept + Lab
6	EFS Storage Classes	Concept + Cert
7	EFS Lifecycle Management & Policies	Concept + Cert
8	EFS Performance Modes & Throughput Modes	Concept + Cert
9	Benefits of EFS	Concept
10	Security & Encryption	Concept + Interview
11	EFS Pricing & Cost Optimization	Concept
12	Lab: Launch 2 EC2 Instances	Lab
13	Lab: Create & Configure EFS	Lab
14	Lab: Mount EFS on Both Instances	Lab
15	Lab: Demonstrate File Sharing	Lab
16	Cleanup Checklist	Lab
17	Assignment — Independent Repeat	Practice

What is Amazon EFS?

EFS = Elastic File System — a fully managed, auto-scaling shared file system that multiple EC2 instances can mount and use at the same time, both reading and writing.

Analogy: Think of EBS as a personal hard drive — it belongs to one laptop only. EFS is like a shared Google Drive folder that your entire team can open simultaneously from different computers, see the same files, and edit them in real time.

Key characteristics:

Uses the NFS 4.1 protocol (Network File System — standard Linux file sharing)
Serverless — no capacity to provision, no servers to manage
Auto-scales — grows from KB to PB automatically, shrinks when files are deleted
Highly available — data replicated across multiple AZs in a Region
Pay-as-you-go — billed per GB actually stored, no pre-provisioning

EFS vs EBS vs S3 — The Big Comparison

This is one of the most commonly tested comparisons in AWS certifications.

Feature	EBS	EFS	S3
Access	Single EC2 instance only	Multiple EC2 instances simultaneously	Internet / API, unlimited clients
Protocol	Block storage	NFS v4.1	HTTP/REST (object storage)
Scope	AZ-specific	Regional (multi-AZ)	Region-wide, globally accessible
Scaling	Manual (resize volume)	Automatic	Unlimited, automatic
Performance	Very high, low latency	High (shared)	Variable
Use Case	OS disks, databases	Shared content, CMS, ML training	Backups, static assets, objects
Cost Model	Provisioned (pay for allocated size)	Usage-based (pay for what's stored)	Usage-based

Simple decision rule:

Need a disk for ONE EC2 instance (like an OS drive)?        → EBS
Need a shared folder MULTIPLE EC2 instances can access?      → EFS
Need to store files accessible from anywhere via internet?   → S3

🎯 Interview Tip: "Can two EC2 instances share an EBS volume?" — Only with io1/io2 Multi-Attach, and even then both must be in the same AZ. EFS is the natural answer when the question is about shared access across multiple instances in different AZs.

EFS Architecture

EFS spans multiple Availability Zones within a Region. Here's how it's structured:

                    Amazon EFS File System
                  (stored redundantly across AZs)
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                      │
      AZ 1                  AZ 2                   AZ 3
   Mount Target          Mount Target           Mount Target
   (ENI + IP)            (ENI + IP)             (ENI + IP)
        │                     │                      │
   EC2 Instance          EC2 Instance           EC2 Instance
   /mnt/efs              /mnt/efs               /mnt/efs

   All instances access the SAME file system via NFS (port 2049)

Mount Target — this is the key piece. It's an ENI (Elastic Network Interface) that EFS creates inside each AZ of your VPC. It's the "doorway" through which EC2 instances in that AZ connect to EFS over NFS.

One mount target per AZ (auto-created when you set up EFS)
Has its own private IP and security group
EC2 instances in that AZ connect to the local mount target for best performance

Mounting — What It Actually Means

Mounting is the process of attaching a file system to a directory on your OS so it behaves like a normal local folder.

Analogy: Mounting EFS is like plugging in a USB drive and having it appear as a folder you can open, save files into, and read from — except this "USB drive" lives in AWS and multiple computers can plug into it at once.

# Before mounting: /mnt/efs is just an empty folder
# After mounting: /mnt/efs IS the EFS file system
sudo mount -t nfs4 -o nfsvers=4.1 <EFS-DNS-Name>:/ /mnt/efs

Once mounted, every ls, cat, echo >, cp command on /mnt/efs is actually reading/writing to the shared EFS file system — not local disk.

Mount Targets & Security Group Configuration

This is the part that trips people up in labs — two separate security groups working together.

The Two Security Groups

EC2 Security Group (lab-ec2-sg)

Inbound:  SSH (22) from My IP
Outbound: NFS (2049) to EFS-SG  [allows EC2 to reach EFS]

EFS Security Group (lab-efs-sg)

Inbound:  NFS (2049) from lab-ec2-sg  [only EC2 instances can reach EFS]
Outbound: Allow all (default)

Why two security groups referencing each other?

This is a common AWS pattern — instead of allowing an IP range, you allow traffic from a specific security group. This means: "Only instances that have the EC2 security group attached are allowed to talk to EFS on port 2049." Clean, scalable, and secure — you never have to know or update IP addresses.

EC2 Instance (has lab-ec2-sg)
        │
        │  NFS traffic, port 2049
        ▼
EFS Mount Target (has lab-efs-sg, allows inbound from lab-ec2-sg)
        │
        ▼
Access Granted ✅

⚠️ Common lab mistake: Creating the EFS file system and trying to mount immediately — it fails because the EFS security group doesn't yet allow NFS traffic from the EC2 security group. You must add that inbound rule first.

Mount Target Requirements

One mount target per AZ in your VPC
Requires a subnet assignment
Has a private IP auto-assigned
DNS-based access — EFS gives you one DNS name that automatically resolves to the correct local mount target for whichever AZ you're connecting from

Persistence Across Reboots

By default, if your EC2 instance restarts, the EFS mount is gone — you'd have to mount it manually again. To make it automatic:

# Add to /etc/fstab
<EFS-DNS-Name>:/ /mnt/efs nfs4 defaults,_netdev 0 0

The _netdev option tells the OS to wait for networking to be ready before attempting the mount — critical since EFS is a network resource, not local disk.

EFS Storage Classes

Storage Class	Use Case	Cost
Standard	Active data, accessed frequently	Higher per GB
Infrequent Access (IA)	Files not accessed often	~50% cheaper per GB + a per-access fee

How it works: Standard is the default — best for files you're actively using. IA is for files that just sit there most of the time (old logs, archived reports) — cheaper to store, but costs a small fee each time you access them. The trade-off only makes sense if access is genuinely rare.

EFS Lifecycle Management & Policies

This is automatic tiering — EFS moves files between Standard and IA based on actual usage, without you lifting a finger.

File created → Standard storage (full price)
        │
        │  Not accessed for X days (your chosen threshold)
        ▼
Auto-moved to Infrequent Access (cheaper)
        │
        │  Someone accesses the file again
        ▼
Auto-moved back to Standard

Configurable transition periods: 7, 14, 30, 60, or 90 days of inactivity (or disabled entirely).

Why this matters: You don't have to manually audit and move files — EFS handles cost optimization automatically based on real access patterns. Set it once during file system creation and forget about it.

🎯 Cert Perspective: "How do you reduce EFS storage cost for rarely accessed files without manual intervention?" → Enable Lifecycle Management with a transition policy to IA.

EFS Performance Modes & Throughput Modes

These are two separate settings that often get confused — performance mode is about latency/concurrency, throughput mode is about data transfer speed.

Performance Modes (set once, cannot change later)

Mode	Optimized For	Use Case
General Purpose (default)	Lowest latency	Web serving, CMS, most workloads — use this unless you have a specific reason not to
Max I/O	Maximum throughput, more concurrent operations	Big data, media processing, highly parallel workloads

Throughput Modes (can change later)

Mode	How it Works	Use Case
Bursting (default)	Throughput scales with file system size, bursts higher when needed	Default choice, cost-effective for most workloads
Provisioned	You specify a guaranteed throughput, independent of storage size	Predictable high-throughput needs regardless of how much is stored
Max I/O	Optimized for highest throughput and ops/second	Heavy parallel processing

🎯 Cert Perspective: Know that Performance Mode is locked at creation, but Throughput Mode can be changed afterward. A scenario like "small file system but needs guaranteed high throughput" points to Provisioned Throughput.

Benefits of EFS

Benefit	What It Means
Scalability	Grows from KB to PB automatically, zero downtime
High Availability	Data redundantly stored across multiple AZs
Durability	Multiple copies protect against component failure
Simple Integration	Standard Linux NFS interface — no app code changes needed
Cost-Optimized	IA storage class cuts costs ~50% for inactive files

Security & Encryption

Network Security:

Security groups control all access (as covered above)
VPC isolation — EFS only accessible within your VPC by default
NFS traffic always on port 2049

Data Encryption:

At rest — via AWS KMS, enable when creating the file system
In transit — via TLS, add -o tls to your mount command

Access Control (3 layers):

File System Policy — IAM-style resource policy on the EFS itself (grant/deny access)
POSIX file permissions — standard Linux permissions (chmod, chown) work exactly as expected
IAM integration — controls who can call EFS API operations (create, delete, modify)

Best practices:

Always enable encryption when creating the file system (can't add it retroactively)
Restrict security groups to required ports only — never open NFS to 0.0.0.0/0
Use VPC security groups for isolation, exactly like the EC2-SG ↔ EFS-SG pattern above

EFS Pricing & Cost Optimization

What you pay for:

Standard Storage — per GB stored
Infrequent Access — ~50% lower per GB, plus a small fee each time data is retrieved

Cost optimization checklist:

✅ Enable Lifecycle Management — auto-move inactive files to IA
✅ Choose sensible transition days based on actual access patterns
✅ Use Bursting Throughput (default) unless you specifically need Provisioned
✅ Monitor usage via CloudWatch metrics
✅ Clean up unused files regularly

🧪 LAB — Full Hands-On Walkthrough

Lab Architecture

            VPC (Default)
        ┌─────────────────────────┐
        │   Availability Zone      │
        │  ┌──────────┐  ┌──────────┐
        │  │   EC2-1   │  │   EC2-2   │
        │  │ /mnt/efs  │  │ /mnt/efs  │
        │  └─────┬────┘  └─────┬────┘
        │        │  NFS (2049)  │
        │        └──────┬───────┘
        │                ▼
        │         Amazon EFS
        │       (Mount Target)
        └─────────────────────────┘

Security Groups:
  EC2-SG  → allows SSH (22) inbound from My IP
  EFS-SG  → allows NFS (2049) inbound from EC2-SG only

Lab Step 1 — Launch 2 EC2 Instances

Repeat this entire process twice (EC2-Instance-1 and EC2-Instance-2):

1. EC2 Console → Launch Instances
2. Name: EC2-Instance-1 (then EC2-Instance-2)
3. AMI: Amazon Linux 2 (HVM) — Free tier eligible
4. Instance type: t2.micro / t3.micro (Free tier)
5. Key Pair: Create new → lab-key-pair → .pem format
6. Network: Default VPC, any subnet, Auto-assign public IP: Enable
7. Security Group: Create lab-ec2-sg → Allow SSH (22) from My IP
8. Storage: 8 GB gp2 root volume (default)
9. Launch Instance — repeat for Instance 2

⚠️ Use the SAME key pair, VPC, and subnet for both instances — keeps the lab simple and ensures they can communicate.

Lab Step 2 — Create and Configure EFS

1. AWS Console → Services → EFS → Create file system
2. Name: lab-efs
3. VPC: Select SAME VPC as your EC2 instances
4. Quick Create (defaults) or Customize
5. EFS auto-creates mount targets in each AZ of the VPC
6. Edit mount target Security Group → create/use lab-efs-sg
7. Add Inbound Rule to lab-efs-sg: NFS (2049) from lab-ec2-sg
8. Create — note the File System ID: fs-xxxxxxxxx

EFS Settings used (defaults):

Performance mode:  General Purpose
Throughput mode:   Bursting
Encryption:        Enabled (KMS)
Storage class:     Standard (multi-AZ redundancy)

Security Group Rule Summary:

Security Group	Direction	Port	Source
lab-ec2-sg	Inbound	22 (SSH)	My IP
lab-efs-sg	Inbound	2049 (NFS)	lab-ec2-sg

Lab Step 3 — Mount EFS on Both Instances

Open two separate SSH terminals — one per instance. Run identical steps on both.

# Step 1: Update packages
sudo yum update -y

# Step 2: Install NFS/EFS utilities
sudo yum install -y amazon-efs-utils
# (or: sudo yum install -y nfs-utils — depending on AMI)

# Step 3: Create mount directory
sudo mkdir efs
# or: sudo mkdir /mnt/efs

# Step 4: Mount EFS
sudo mount -t efs -o tls fs-012f61809a9d6e846:/ /efs
# or using DNS name:
sudo mount -t nfs4 -o nfsvers=4.1 <EFS-DNS-Name>:/ /mnt/efs

# Step 5: Verify the mount
df -h
# or
mount | grep efs

⚠️ If the mount fails on the first attempt — this is expected if you haven't yet added the NFS inbound rule to the EFS security group. Add it, then retry the mount command. This is intentional in the lab flow to show why the security group rule matters.

Lab Step 4 — Demonstrate File Sharing (The Core Proof)

On EC2 Instance 1 (Writer):

cd /mnt/efs

# Write a file
echo 'Hello from EC2 Instance 1!' | sudo tee shared-file.txt

# Append a timestamp
echo "Timestamp: $(date)" | sudo tee -a shared-file.txt

# Create a folder and copy a file into it
sudo mkdir /mnt/efs/shared-folder
sudo cp /etc/hostname /mnt/efs/shared-folder/host1.txt

On EC2 Instance 2 (Reader) — same EFS mount:

cd /mnt/efs

# List files — see what Instance 1 wrote, instantly
ls -la /mnt/efs
# Output: shared-file.txt  shared-folder/

# Read the file Instance 1 created
cat shared-file.txt
# Output:
# Hello from EC2 Instance 1!
# Timestamp: Mon Jan 01 10:00:00 UTC

# Write back from Instance 2
echo 'Reply from Instance 2!' | sudo tee -a shared-file.txt

Verify on EITHER instance:

cat /mnt/efs/shared-file.txt
# Hello from EC2 Instance 1!
# Timestamp: Mon Jan 01 10:00:00 UTC
# Reply from Instance 2!

What this proves:

Both instances read/write the same file system simultaneously
Changes from one instance are instantly visible on the other
No manual file transfer, sync, or copying needed — this is the entire value of EFS

Permissions Note

# Grant full access to all users on the mount
sudo chmod 777 /mnt/efs

# Now you can write without sudo
echo 'No sudo needed!' > /mnt/efs/test.txt
cat /mnt/efs/test.txt

🧹 Cleanup Checklist (Don't Skip This!)

✅ Unmount EFS on both instances: sudo umount /mnt/efs
✅ Terminate both EC2 instances
✅ Delete EFS file system (lab-efs)
✅ Delete security groups: lab-ec2-sg, lab-efs-sg
✅ Delete key pair if no longer needed

⚠️ EFS bills per GB stored — even small test files left running add up over time if forgotten. Always clean up after labs.

📝 Assignment — Independent Repeat

Complete this without step-by-step guidance, using everything from the lab above:

1.  Launch EC2 #1 — Amazon Linux 2, t2.micro, new key pair, public IP enabled
2.  Launch EC2 #2 — same config, same key pair, same VPC/subnet
3.  Create EFS — name: my-efs, default VPC, default settings
4.  Configure SGs — EC2 SG: SSH (22). EFS SG: NFS (2049) from EC2 SG
5.  Mount on EC2 #1 — install nfs-utils, mkdir /mnt/efs, mount EFS
6.  Mount on EC2 #2 — same mount steps
7.  Verify Mounts — run df -h on both, confirm EFS appears
8.  Write from EC2 #1 — create a file with your name at /mnt/efs/yourname.txt
9.  Read from EC2 #2 — cat /mnt/efs/yourname.txt, confirm visible
10. Write from EC2 #2 — append a second line to the same file
11. Cleanup — unmount, terminate instances, delete EFS and SGs

Submission deliverables (screenshots):

EC2 Instances list — both Running
EFS file system page — showing created file system
df -h output on Instance 1 — /mnt/efs mounted
df -h output on Instance 2 — /mnt/efs mounted
Terminal — file written on Instance 1
Terminal — same file read from Instance 2

Grading breakdown: EC2 launch (25 pts) · EFS + SG config (25 pts) · Mounted on both (25 pts) · File sharing both ways (25 pts)

⚡ Quick Revision

WHAT IS EFS
  Fully managed, shared, auto-scaling NFS file system
  Multiple EC2 instances mount it simultaneously
  Protocol: NFS v4.1, Port: 2049

EFS vs EBS vs S3
  EBS = single instance, block storage, AZ-locked
  EFS = multiple instances, NFS, Region-wide (multi-AZ)
  S3  = internet/API access, object storage, unlimited

ARCHITECTURE
  Mount Target = ENI in each AZ, the doorway to EFS
  One per AZ, auto-created, has its own private IP + SG

SECURITY GROUP PATTERN
  EC2-SG  → outbound NFS to EFS-SG
  EFS-SG  → inbound NFS (2049) FROM EC2-SG only
  Never open NFS to 0.0.0.0/0

STORAGE CLASSES
  Standard = frequent access, higher cost
  IA       = infrequent access, ~50% cheaper + access fee

LIFECYCLE POLICY
  Auto-moves files Standard → IA after X days inactive (7/14/30/60/90)
  Auto-moves back to Standard when accessed again

PERFORMANCE MODES (set once)
  General Purpose = default, lowest latency, most workloads
  Max I/O = max throughput, more concurrency, big data

THROUGHPUT MODES (changeable)
  Bursting     = default, scales with FS size
  Provisioned  = fixed guaranteed throughput
  Max I/O      = highest throughput/ops

ENCRYPTION
  At rest  → KMS (enable at creation)
  In transit → TLS (-o tls in mount command)

MOUNT COMMAND
  sudo mount -t efs -o tls fs-xxxx:/ /efs
  or
  sudo mount -t nfs4 -o nfsvers=4.1 <EFS-DNS>:/ /mnt/efs

PERSISTENCE
  Add to /etc/fstab with _netdev option to survive reboot

💼 Interview Questions

Q1: What is the main difference between EFS and EBS?
EBS is block storage attached to a single EC2 instance within one AZ — like a personal hard drive. EFS is a shared NFS file system that multiple EC2 instances across multiple AZs can mount and access simultaneously — like a shared network drive.

Q2: Why does an EFS mount initially fail in the lab even though the file system was created successfully?
The EFS security group doesn't yet have an inbound rule allowing NFS (port 2049) traffic from the EC2 security group. Until that rule is added, EC2 instances cannot reach the mount target, so the mount command times out or fails.

Q3: How does EFS achieve high availability?
EFS automatically creates a mount target in each Availability Zone of your VPC and stores data redundantly across multiple AZs within the Region — so it tolerates an AZ failure without data loss or downtime.

Q4: What's the difference between EFS Performance Mode and Throughput Mode?
Performance Mode (General Purpose or Max I/O) is set once at creation and determines latency characteristics — it can't be changed later. Throughput Mode (Bursting, Provisioned, or Max I/O) controls how much data transfer capacity you get and can be changed after creation.

Q5: A company has files that are rarely accessed after 30 days but must remain available. How would you optimize EFS costs?
Enable EFS Lifecycle Management with a transition policy set to 30 days. Files inactive for 30 days automatically move to Infrequent Access storage class at roughly 50% lower storage cost. If accessed again, they automatically move back to Standard.

Q6: Why do EC2-SG and EFS-SG reference each other instead of using IP addresses?
Referencing security groups instead of IPs is more secure and scalable — it means "only resources carrying this specific security group can connect," regardless of how many instances exist or what their IPs are. No manual IP management needed as you scale.

🔬 Practice Tasks

Complete the Assignment above independently (full EC2 + EFS lab without referring to lab steps).
After mounting EFS on one instance, deliberately remove the NFS inbound rule from the EFS security group, then try writing a new file. Observe what happens. Re-add the rule and confirm it works again.
Enable Lifecycle Management on your EFS with a 7-day transition to IA. Create a file, wait (or simulate), and check the EFS Console to see which storage class it reports.
Add your EFS mount to /etc/fstab on one instance. Reboot the instance and confirm the mount persists automatically without running the mount command again.
Compare: create a 1GB file directly on EBS root volume vs on the EFS mount. Time how long a cp operation takes on each, and note any latency differences.

AWS Session 5 — EFS Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer