DEV Community: Tejas Shinkar

AWS S3 Deep Dive — Objects, Encryption, Bucket Policies & Everything In Between

Tejas Shinkar — Mon, 20 Jul 2026 13:27:36 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session goes deep into S3 — the most widely used AWS service and a guaranteed topic in every AWS certification and DevOps interview.

📋 Topics Covered

#	Topic	Type
1	Why S3 Matters + Quick Recap	Concept
2	JSON & YAML — Why They Matter in AWS	Concept + Interview
3	API Calls in AWS (GET, PUT, DELETE)	Concept + DevOps
4	Region Selection — Why It Matters for S3	Concept + Cert
5	S3 Buckets — Rules & Constraints	Concept + Lab
6	S3 Objects — What They Actually Are	Concept + Cert
7	S3 Object URL Structure	Concept + Lab
8	S3 Storage Classes	Concept + Cert
9	S3 Versioning	Concept + Lab
10	S3 Encryption — SSE-S3, SSE-KMS, SSE-C, CSE	Concept + Cert
11	S3 Bucket Keys — KMS Cost Optimization	Concept + Cert
12	Bucket Policies — Ownership & ACL Settings	Concept + Interview
13	S3 Access Control — Policies, ACLs, Public Access	Concept + Cert
14	Explicit Deny — The Override Rule	Concept + Interview
15	AWS CloudTrail + S3 Audit Logging	Concept + DevOps
16	VPC — Brief Introduction	Concept
17	S3 CLI Commands	Practical
18	boto3 — S3 with Python	Practical
19	Interview Questions	Interview
20	Practice Tasks	Practice

Why S3 Matters

S3 (Simple Storage Service) is arguably the most important AWS service to understand — it appears in almost every architecture, every certification, and every DevOps workflow. Backups, static websites, data lakes, CI/CD artifacts, ML training data, log archives — S3 underpins all of it.

Before diving into encryption and policies, two foundational things from this session need to be covered first: JSON/YAML code and API calls — because everything in S3 (and AWS) is built on top of these.

JSON & YAML — Why Interviewers Ask You to Write Them

In real organizations, nobody clicks through the AWS Console to create resources. Everything is written as code — JSON or YAML — and executed via CLI, CloudFormation, or Terraform. Interviewers increasingly ask candidates to write infrastructure code on the spot (sometimes called "vibe coding").

JSON (JavaScript Object Notation):

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "MyEC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": "ami-0abcdef1234567890",
        "InstanceType": "t2.micro",
        "KeyName": "my-key-pair",
        "Tags": [
          { "Key": "Name", "Value": "MyServer" }
        ]
      }
    }
  }
}

YAML (Yet Another Markup Language) — same thing, cleaner to read:

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-0abcdef1234567890
      InstanceType: t2.micro
      KeyName: my-key-pair
      Tags:
        - Key: Name
          Value: MyServer

Key differences:

	JSON	YAML
Syntax	Curly braces `{}`, quotes `""`, commas	Indentation, colons `:`, cleaner
Readability	Verbose, harder to read	Clean, easier to write by hand
Comments	❌ Not supported	✅ Supported with `#`
Used in AWS for	API responses, Lambda, IAM policies	CloudFormation, SAM, CodePipeline
Error-prone	Missing comma or brace	Wrong indentation

S3 Bucket Policy in JSON (what you'll write in real interviews):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPublicRead",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket-name/*"
    }
  ]
}

IAM Policy in YAML (CloudFormation):

Type: AWS::IAM::Policy
Properties:
  PolicyName: S3ReadPolicy
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - s3:GetObject
          - s3:ListBucket
        Resource:
          - arn:aws:s3:::my-bucket
          - arn:aws:s3:::my-bucket/*

🎯 Interview tip: Practice writing a basic S3 bucket policy and EC2 IAM role in both JSON and YAML without looking things up. Interviewers genuinely ask for this now.

API Calls in AWS — How Everything Actually Works

Every click in the AWS Console, every CLI command, every boto3 call — all of it is ultimately an HTTP API call under the hood. Understanding this is essential for DevOps and automation work.

AWS uses standard HTTP methods:

Method	Purpose	S3 Example
`GET`	Read / retrieve data	Download an object, list buckets
`PUT`	Create or replace	Upload an object, create a bucket
`POST`	Create or trigger	Initiate multipart upload
`DELETE`	Remove	Delete an object or bucket
`HEAD`	Get metadata only	Check if object exists without downloading

EC2 → S3 interaction via API:

EC2 Instance
    │
    │  PUT /my-bucket/backup.zip  (upload)
    │  GET /my-bucket/config.json (download)
    │  DELETE /my-bucket/old.log  (delete)
    ▼
S3 API Endpoint: https://s3.ap-south-1.amazonaws.com
    │
    ▼
S3 Bucket (my-bucket)

In practice, your application on EC2 uses the AWS SDK (boto3 in Python, or the Java/Node SDK) which converts your code calls into these signed HTTP API requests automatically.

AWS Signature Version 4 — all API calls must be signed with your credentials so AWS can verify identity. The SDK handles this for you when using IAM roles (recommended) or access keys (not recommended).

🎯 DevOps relevance: When you write a Python script with boto3 to copy files to S3, you're making PUT API calls. When your Lambda function reads from DynamoDB, it's making GET API calls. The console is just a visual wrapper over the same APIs.

Region Selection — Why It Matters for S3

S3 is a global service in name, but buckets are created in a specific Region. Choosing the right region affects:

Factor	Impact on S3
Performance	Store data in Region closest to your users/EC2 → lower latency
Cost	S3 pricing varies by Region (US East is typically cheapest)
Data Transfer	Transferring data between Regions costs money — keep S3 and EC2 in same Region
Compliance / Regulatory	GDPR (EU), RBI (India) may legally require data in specific regions
Disaster Recovery	Cross-Region Replication (CRR) copies to another Region for DR

💡 Practical rule: Always create your S3 bucket in the same Region as the EC2 instances or services that will use it. Cross-region data transfer is charged per GB and adds latency.

S3 Buckets — Rules & Constraints

A bucket is the container for your objects in S3. Think of it like a top-level folder — except with globally unique naming rules and no nesting of buckets inside buckets.

Bucket Naming Rules

✅ Must be globally unique — no two buckets in the world can share a name
✅ 3 to 63 characters long
✅ Lowercase letters, numbers, and hyphens only
✅ Must start with a letter or number
✅ Must not start with "xn--" (reserved)
✅ Must not end with "-s3alias"
❌ No uppercase letters
❌ No underscores
❌ No IP address format (e.g., 192.168.1.1)
❌ No spaces

Good names: tejas-dev-backups, company-prod-logs-2026, my-static-site-assets

Bad names: My_Bucket, 123.456.789.0, TejasBucket

Key Bucket Facts

One Region per bucket — created in a specific Region, stays there
No size limit on total bucket storage — scales infinitely
5 TB max per individual object — single file can't exceed 5TB
Soft limit of 100 buckets per account (can request increase to 1000)
No nested buckets — you cannot put a bucket inside a bucket
Bucket names in URLs — because names are globally unique, they're used directly in URLs

S3 Objects — What They Actually Are

An object is any file stored in S3. Unlike a traditional file system, S3 is flat — there are no real folders (just prefixes that look like folders).

Object Components

Component	Description
Key	The full "path" of the object. e.g., `reports/2026/january/sales.csv`
Value	The actual file content (bytes)
Version ID	Unique ID if versioning is enabled
Metadata	Key-value pairs about the object (content-type, custom tags)
ETag	MD5 hash of the object (used to verify integrity)
Storage Class	Standard, IA, Glacier, etc.

The "Folder" Illusion

Bucket: my-data
Objects stored as:
  reports/2026/sales.csv    ← key includes the "path"
  reports/2026/revenue.csv
  images/logo.png
  images/banner.jpg
  config.json

There are NO real folders. "reports/2026/" is just a prefix.
The console shows them as folders for readability — but it's flat storage underneath.

Object Size Limits

Upload type	Size limit	When to use
Single PUT upload	Up to 5 GB	Small to medium files
Multipart upload	Up to 5 TB	Files larger than 100 MB (recommended)
Minimum part size	5 MB	For multipart upload chunks

✅ Best practice: Use multipart upload for any file over 100 MB. It's faster (parallel chunks), more reliable (resume on failure), and required above 5 GB.

S3 Object URL Structure

Every object in S3 has a predictable, deterministic URL. Understanding the format is important for linking to files, debugging access issues, and building applications.

URL Formats

Path-style URL (older, being deprecated):

https://s3.<region>.amazonaws.com/<bucket-name>/<object-key>
https://s3.ap-south-1.amazonaws.com/tejas-dev/reports/sales.csv

Virtual-hosted-style URL (current standard):

https://<bucket-name>.s3.<region>.amazonaws.com/<object-key>
https://tejas-dev.s3.ap-south-1.amazonaws.com/reports/sales.csv

Breaking it down:

https://tejas-dev.s3.ap-south-1.amazonaws.com/reports/2026/sales.csv
        │          │   │                        │
        bucket     │   region                   object key (full path)
                   S3 service

URL Validity

An S3 object URL is only valid while:

The object actually exists in the bucket
The object is publicly accessible (or you're using a pre-signed URL with a valid expiry)

If you delete the object, the URL returns a 404 — the URL structure stays the same but there's nothing at that address.

Pre-Signed URLs

For private objects, you can generate a pre-signed URL — a temporary link with an expiry time that grants access without making the object public.

import boto3
s3 = boto3.client('s3')

# Generate a URL valid for 1 hour
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-private-bucket', 'Key': 'secret-file.pdf'},
    ExpiresIn=3600  # seconds
)
print(url)
# https://my-private-bucket.s3.ap-south-1.amazonaws.com/secret-file.pdf?X-Amz-Signature=...&X-Amz-Expires=3600

🎯 Common use case: Share a download link for a private file with a customer for 24 hours — pre-signed URL with ExpiresIn=86400. Link expires automatically.

S3 Storage Classes

S3 isn't one-size-fits-all for storage. Different classes trade cost vs access speed vs retrieval time.

Storage Class	Use Case	Retrieval	Cost
S3 Standard	Frequently accessed data	Milliseconds	Highest
S3 Intelligent-Tiering	Unknown or changing access patterns	Milliseconds	Moderate (auto-tiered)
S3 Standard-IA	Infrequent access, needs fast retrieval	Milliseconds	Lower storage, retrieval fee
S3 One Zone-IA	Infrequent, reproducible data, one AZ only	Milliseconds	Lower than Standard-IA
S3 Glacier Instant	Archive, accessed once a quarter	Milliseconds	Very low
S3 Glacier Flexible	Archive, minutes-to-hours retrieval ok	Minutes to hours	Very low
S3 Glacier Deep Archive	Long-term archive, accessed once a year	Up to 12 hours	Cheapest

Simple decision guide:

Serving to users daily?              → Standard
Don't know access pattern?           → Intelligent-Tiering (auto-manages)
Accessed once a month, need fast?    → Standard-IA
Old logs, rarely accessed?           → Glacier Instant Retrieval
Legal archive, won't need for years? → Glacier Deep Archive

Lifecycle Policies (Auto-Tiering)

Just like EFS, you can set rules to automatically move objects between storage classes:

Day 0    → Upload to Standard
Day 30   → Auto-move to Standard-IA
Day 90   → Auto-move to Glacier Instant
Day 365  → Auto-move to Glacier Deep Archive
Day 2555 → Auto-delete (7 years, for compliance)

🎯 Cert tip: "How do you minimize S3 storage costs for logs that need to be kept for 7 years but rarely accessed after 30 days?" → Lifecycle policy transitioning to Glacier.

S3 Versioning

When versioning is enabled, S3 keeps every version of every object — overwrites create a new version instead of replacing, and deletes add a "delete marker" instead of actually deleting.

Upload file.txt (v1)      → Version ID: aaa111
Upload file.txt (v2)      → Version ID: bbb222  ← current version
Upload file.txt (v3)      → Version ID: ccc333  ← current version

GET file.txt              → returns v3 (current)
GET file.txt?versionId=aaa111 → returns v1
DELETE file.txt           → adds delete marker, v3 still exists

Why versioning matters:

Accidental overwrite? Restore previous version instantly
Accidental delete? Remove the delete marker to recover
Required for Cross-Region Replication
Required for S3 Object Lock (compliance)

⚠️ Cost warning: Versioning stores every version of every file. An object updated 100 times = 100 stored copies. Enable Lifecycle Policies to expire old versions, otherwise storage costs grow silently.

S3 Encryption — Four Methods

Encryption in S3 has two dimensions: at rest (stored on disk) and in transit (moving over the network). Transit is always HTTPS — the four methods below are all about at rest.

SSE-S3 (Server-Side Encryption with S3-Managed Keys)

AWS manages everything — keys are created, rotated, and used by S3 automatically
Enabled by default on all new buckets (since Jan 2023)
Zero configuration needed
AES-256 encryption
You have no control over the keys

Upload file → S3 auto-encrypts with its own key → stored encrypted
Download → S3 auto-decrypts → you get the file
You never see or touch the key

Use when: You need encryption but don't have compliance requirements about key control.

SSE-KMS (Server-Side Encryption with AWS KMS Keys)

Used mostly

Upload flow:

You upload file.pdf
      ↓
S3 sends request to AWS KMS: "Give me a data encryption key"
      ↓
KMS generates a data key (encrypted + plaintext versions)
      ↓
S3 uses plaintext key to encrypt your file
      ↓
S3 stores: encrypted file + encrypted key (plaintext key discarded)
      ↓
Your file sits on disk: fully encrypted ✅

Download flow:

You request file.pdf
      ↓
S3 checks your IAM permissions (must have s3:GetObject)
      ↓
S3 checks your KMS permissions (must have kms:Decrypt)
      ↓
S3 sends encrypted key to KMS: "Decrypt this"
      ↓
KMS decrypts the key → returns plaintext key
      ↓
S3 uses plaintext key to decrypt your file
      ↓
You receive the original file ✅

Why SSE-KMS over SSE-S3?

You control the KMS key (create, rotate, disable, delete)
Full audit trail — every KMS key usage logged in CloudTrail
Granular access control — you can restrict who can use the key via KMS key policy
Required by many compliance standards (HIPAA, PCI-DSS, FedRAMP)

Downside: Every object upload/download = one KMS API call. KMS has a cost per API call and rate limits.

SSE-C (Server-Side Encryption with Customer-Provided Keys)

You provide your own encryption key with each request
AWS uses it to encrypt/decrypt, but never stores your key
You are 100% responsible for key management

Upload: You send (file + your-key) → S3 encrypts → discards your-key
Download: You send (object-key + your-key) → S3 decrypts → returns file
Lose your key = lose your data permanently (no recovery)

Use when: You have strict regulatory requirements that AWS must never hold your encryption key even temporarily.

CSE (Client-Side Encryption)

You encrypt the file before uploading it to S3
S3 just stores whatever bytes you send — it has no idea the content is encrypted
You manage all encryption/decryption on your side

Your app: file → encrypt locally → send encrypted bytes → S3 stores them
Your app: download encrypted bytes → decrypt locally → original file
AWS sees only ciphertext — never the original data

Use when: Maximum data confidentiality — even AWS employees cannot read your data.

Encryption Comparison

Method	Who manages key	AWS sees plaintext?	Audit trail	Use when
SSE-S3	AWS (automatic)	Internally yes	Basic	Default, no compliance needs
SSE-KMS	You (via KMS)	Internally yes	Full (CloudTrail)	Compliance, audit requirements
SSE-C	You	Never	None	Strict key control
CSE	You	Never	None	Maximum confidentiality

S3 Bucket Keys — Reducing KMS Costs by 99%

With SSE-KMS, every single object upload/download triggers a KMS API call. For buckets with millions of objects, this generates millions of KMS calls — which have both a per-call cost and API rate limits.

S3 Bucket Keys solve this:

Without Bucket Key:

100 object uploads → 100 KMS API calls → $$$

With Bucket Key enabled:

S3 generates one bucket-level data key from KMS
That key is cached in S3 for a time period
100 object uploads → 1 KMS API call + 99 local operations → $ (99% fewer calls)

Analogy: Instead of going to the bank (KMS) to get cash for every purchase, you withdraw a stack of cash once and use it locally for many purchases.

Reduces KMS API costs by up to 99%
Reduces KMS request rate (avoids throttling)
Transparent — encryption strength is unchanged
Enable it when creating the bucket or in bucket properties

🎯 Cert tip: "A company uses SSE-KMS on an S3 bucket with millions of daily uploads and is hitting KMS rate limits. What's the fix?" → Enable S3 Bucket Keys.

AWS CloudTrail + S3 Audit Logging

CloudTrail records every API call made to AWS services — who did what, when, from where. For S3 with KMS encryption, it becomes your full audit trail.

What CloudTrail logs for S3 + KMS:

Who:    IAM user/role that accessed the object
What:   s3:GetObject, s3:PutObject, kms:Decrypt, kms:GenerateDataKey
When:   Timestamp of the request
Where:  Source IP address
Result: Success or failure (and why if failed)

Two types of S3 events in CloudTrail:

Management Events — bucket-level operations (CreateBucket, DeleteBucket, PutBucketPolicy) — logged by default
Data Events — object-level operations (GetObject, PutObject, DeleteObject) — must be explicitly enabled (generates high volume)

S3 Server Access Logging (separate from CloudTrail):

Detailed request logs written directly to another S3 bucket
Every HTTP request to the bucket: requester IP, request type, response code, bytes transferred
Good for access pattern analysis and billing investigation

🎯 DevOps/Security use case: Enable CloudTrail data events for your S3 bucket to detect unauthorized access. If someone outside your organization downloads a sensitive file, CloudTrail has the full record — user, IP, timestamp, object key.

Bucket Policies — Object Ownership & ACL Settings

When multiple AWS accounts are involved (or when files are uploaded by external parties), there are three ownership models that control who owns uploaded objects and how ACLs work.

Three Object Ownership Settings

1. Bucket Owner Enforced (Recommended)

All objects in the bucket are always owned by the bucket owner
ACLs are completely disabled
All access must be controlled via bucket policies or IAM policies only
Simplest and most secure model

External account uploads file → bucket owner automatically owns it
No ACL confusion, no cross-account ownership issues

2. Bucket Owner Preferred

Bucket owner takes ownership IF the uploader includes bucket-owner-full-control ACL
ACLs are still enabled
Gives flexibility while nudging toward central ownership

External account uploads with bucket-owner-full-control ACL
→ bucket owner gets ownership ✅
External account uploads without that ACL
→ external account retains ownership ⚠️

3. Object Writer

The account that uploads an object owns it (legacy default behavior)
Can cause situations where bucket owner cannot read/delete objects in their own bucket
ACLs enabled

✅ Best practice: Always use Bucket Owner Enforced for new buckets. It's simpler, removes ACL complexity, and avoids cross-account ownership confusion that has caused real security incidents.

S3 Access Control — Full Picture

S3 has multiple layers of access control that work together:

Request to access S3 object
        │
        ▼
1. IAM Policy — does the requester's IAM user/role allow this?
        │
        ▼
2. Bucket Policy — does the bucket policy allow this request?
        │
        ▼
3. S3 Block Public Access — is public access blocked at account/bucket level?
        │
        ▼
4. ACL (if enabled) — does the object/bucket ACL permit access?
        │
        ▼
Access Granted or Denied

All applicable policies are evaluated together. An explicit Deny at ANY layer = denied, regardless of allows elsewhere.

Bucket Policy

A JSON document attached to the bucket that defines who can access it and what they can do:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowEC2Access",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:role/EC2-S3-Role"
      },
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyDeleteForEveryone",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

S3 Block Public Access (4 Settings)

AWS added this specifically to prevent accidental data exposure — one of the most common cloud security incidents.

Setting	What it blocks
Block public ACLs	Rejects PUT requests with public ACLs
Ignore public ACLs	Ignores existing public ACLs
Block public bucket policies	Rejects bucket policies that grant public access
Restrict public bucket policies	Restricts public access even if bucket policy allows it

✅ Best practice: Enable all 4 Block Public Access settings at the account level. Explicitly disable only for specific buckets that genuinely need public access (e.g., static website hosting).

Explicit Deny — The Override Rule

This is one of the most important concepts in all of AWS security — and it applies everywhere, not just S3.

The rule:

Explicit Deny ALWAYS wins over any Allow, from any source

Policy evaluation order:

1. Is there an Explicit Deny anywhere? → DENIED (done, nothing else checked)
2. Is there an Explicit Allow? → ALLOWED
3. Neither? → DENIED (implicit deny is the default)

Why this matters in practice:

Scenario: User has AdministratorAccess (allows everything)
          SCP says: Deny s3:DeleteBucket

Result: User CANNOT delete S3 buckets — despite having AdministratorAccess
The Deny wins. Always.

Real example in a bucket policy:

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": "arn:aws:s3:::critical-data/*",
  "Condition": {
    "Bool": {
      "aws:SecureTransport": "false"
    }
  }
}

This denies all S3 actions if the request isn't using HTTPS — even if the user has full S3 permissions elsewhere. The Deny wins.

🎯 Interview tip: "A user has S3FullAccess but can't delete objects from a specific bucket. Why?" → There's likely an explicit Deny somewhere — in a bucket policy, SCP, or permission boundary. Explicit Deny always overrides explicit Allow.

VPC — Brief Introduction

VPC = Virtual Private Cloud — your own logically isolated private network inside AWS.

Think of it as buying a plot of land inside a large city (AWS). Within your plot, you decide the layout — which areas are public-facing, which are private, what gets connected to the internet.

Why it's relevant to S3 here: By default, S3 is accessed over the public internet (via HTTPS). For security, you can set up a VPC Endpoint for S3 — a private connection from your VPC directly to S3 without traffic leaving the AWS network.

Without VPC Endpoint:
EC2 → public internet → S3 (traffic exits AWS network)

With VPC Endpoint (Gateway type):
EC2 → VPC Endpoint → S3 (traffic stays inside AWS backbone)
     No internet gateway needed
     More secure, no data transfer costs for S3

VPC will be covered in depth in a dedicated session. For now: VPC = your private network, and S3 can be accessed privately through it via Gateway Endpoints.

S3 CLI Commands

# List all buckets
aws s3 ls

# List objects in a bucket
aws s3 ls s3://my-bucket/
aws s3 ls s3://my-bucket/reports/ --recursive

# Create a bucket
aws s3 mb s3://my-new-bucket-name --region ap-south-1

# Upload a file
aws s3 cp local-file.txt s3://my-bucket/uploads/file.txt

# Upload entire folder
aws s3 cp ./my-folder s3://my-bucket/my-folder/ --recursive

# Download a file
aws s3 cp s3://my-bucket/file.txt ./local-file.txt

# Sync local folder to S3 (only uploads new/changed files)
aws s3 sync ./local-folder s3://my-bucket/folder/

# Delete an object
aws s3 rm s3://my-bucket/old-file.txt

# Delete all objects in a folder
aws s3 rm s3://my-bucket/old-folder/ --recursive

# Delete a bucket (must be empty first)
aws s3 rb s3://my-bucket

# Delete bucket and all contents
aws s3 rb s3://my-bucket --force

# Copy between buckets (cross-region)
aws s3 cp s3://source-bucket/file.txt s3://dest-bucket/file.txt \
  --source-region ap-south-1 --region us-east-1

# Set storage class on upload
aws s3 cp file.txt s3://my-bucket/ --storage-class STANDARD_IA

# Generate pre-signed URL (valid 1 hour)
aws s3 presign s3://my-bucket/private-file.pdf --expires-in 3600

# Put bucket policy
aws s3api put-bucket-policy \
  --bucket my-bucket \
  --policy file://policy.json

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled

boto3 — S3 with Python

import boto3

s3 = boto3.client('s3', region_name='ap-south-1')
s3_resource = boto3.resource('s3')

# ── List buckets ────────────────────────────────────────────
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(f"Bucket: {bucket['Name']} | Created: {bucket['CreationDate']}")

# ── Create bucket ───────────────────────────────────────────
s3.create_bucket(
    Bucket='my-new-bucket-2026',
    CreateBucketConfiguration={'LocationConstraint': 'ap-south-1'}
)

# ── Upload file ─────────────────────────────────────────────
s3.upload_file('local-file.txt', 'my-bucket', 'uploads/file.txt')

# ── Upload with SSE-KMS encryption ─────────────────────────
s3.upload_file(
    'sensitive.pdf',
    'my-bucket',
    'secure/sensitive.pdf',
    ExtraArgs={
        'ServerSideEncryption': 'aws:kms',
        'SSEKMSKeyId': 'arn:aws:kms:ap-south-1:123456789:key/abc-123'
    }
)

# ── Download file ───────────────────────────────────────────
s3.download_file('my-bucket', 'uploads/file.txt', 'downloaded-file.txt')

# ── List objects in bucket ──────────────────────────────────
response = s3.list_objects_v2(Bucket='my-bucket', Prefix='reports/')
for obj in response.get('Contents', []):
    print(f"Key: {obj['Key']} | Size: {obj['Size']} bytes")

# ── Delete object ───────────────────────────────────────────
s3.delete_object(Bucket='my-bucket', Key='old-file.txt')

# ── Generate pre-signed URL ─────────────────────────────────
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'private-doc.pdf'},
    ExpiresIn=3600
)
print(f"Share this URL (valid 1 hour): {url}")

# ── Apply bucket policy ─────────────────────────────────────
import json
policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
        ],
        "Condition": {"Bool": {"aws:SecureTransport": "false"}}
    }]
}
s3.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))

⚡ Quick Revision

JSON vs YAML
  JSON = braces/quotes/commas, API responses, IAM policies
  YAML = indentation/colons, CloudFormation, SAM
  Both used for infrastructure-as-code — know both

API CALLS
  GET    = read/retrieve
  PUT    = create/replace
  POST   = create/trigger
  DELETE = remove
  HEAD   = metadata only

S3 OBJECT URL
  https://<bucket>.s3.<region>.amazonaws.com/<key>
  Valid only while object exists and access is permitted
  Pre-signed URL = temporary access for private objects

STORAGE CLASSES (cost vs speed)
  Standard          → frequent access, fastest, most expensive
  Intelligent-Tiering → unknown patterns, auto-optimizes
  Standard-IA       → monthly access, fast retrieval fee
  Glacier Instant   → quarterly access, instant retrieval
  Glacier Deep      → annual access, 12hr retrieval, cheapest

ENCRYPTION
  SSE-S3  → AWS manages key, default, zero config
  SSE-KMS → you control key via KMS, audit trail, compliance
  SSE-C   → you provide key per request, AWS never stores it
  CSE     → you encrypt before upload, AWS sees ciphertext only

S3 BUCKET KEYS
  Reduces KMS API calls by 99%
  One bucket-level key cached, used for multiple objects
  Enable for high-volume SSE-KMS buckets

BUCKET OWNERSHIP
  Owner Enforced  = ACLs off, bucket owner owns all (use this)
  Owner Preferred = owner gets ownership if ACL included
  Object Writer   = uploader owns (legacy, avoid)

EXPLICIT DENY
  Always overrides any Allow, from any source
  Evaluation: Explicit Deny → Explicit Allow → Implicit Deny

CLOUDTRAIL
  Logs every API call: who, what, when, where, result
  Data events (object-level) must be explicitly enabled
  Use for security auditing of S3 + KMS access

💼 Interview Questions

Q1: What is the difference between an S3 bucket and an S3 object?
A bucket is the container — like a top-level folder — created in a specific Region with a globally unique name. An object is the actual file stored inside the bucket, uniquely identified by its key (path). The bucket holds the object; the object is the data.

Q2: What makes an S3 object URL unique and when does it stop working?
The URL is formed from the bucket name, region, and object key — all of which are unique. It stops working when the object is deleted or when access permissions are revoked. For private objects, you need a pre-signed URL with an expiry time to share temporary access.

Q3: What is SSE-KMS and how does it work?
SSE-KMS uses AWS Key Management Service for encryption. On upload, S3 requests a data key from KMS, encrypts the file with the plaintext key, and stores both the encrypted file and the encrypted key (discarding the plaintext key). On download, S3 verifies IAM and KMS permissions, asks KMS to decrypt the encrypted key, then decrypts and returns the file. You control the KMS key and every usage is logged in CloudTrail.

Q4: What are S3 Bucket Keys and why would you enable them?
S3 Bucket Keys are a bucket-level data key cached in S3, used to encrypt multiple objects instead of calling KMS for every single operation. This reduces KMS API calls by up to 99%, significantly reducing cost and avoiding KMS rate limit throttling — essential for high-throughput S3 buckets using SSE-KMS.

Q5: What is the difference between SSE-KMS and SSE-C?
SSE-KMS means AWS generates and manages the data keys inside KMS — you control the master key but AWS handles the actual encryption. SSE-C means you provide your own encryption key with every request — AWS uses it and immediately discards it, never storing it. If you lose your SSE-C key, your data is gone forever.

Q6: An IAM user has S3FullAccess but cannot delete objects from a bucket. Why?
There's an explicit Deny somewhere overriding the Allow. It could be in the bucket policy, an SCP at the Organizations level, or a permission boundary. Explicit Deny always wins over any Allow, from any source.

Q7: What is Bucket Owner Enforced and why is it recommended?
It disables ACLs entirely and ensures the bucket owner always owns all objects regardless of who uploaded them. It's the simplest and most secure model — no ACL confusion, no cross-account ownership issues, and access is controlled purely through bucket policies and IAM.

Q8: Write a bucket policy that denies all HTTP (non-HTTPS) access.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::my-bucket",
      "arn:aws:s3:::my-bucket/*"
    ],
    "Condition": {
      "Bool": {"aws:SecureTransport": "false"}
    }
  }]
}

🔬 Practice Tasks

Create and configure a bucket: Create an S3 bucket with versioning enabled, Block Public Access on, and a lifecycle policy that moves objects to Standard-IA after 30 days and Glacier after 90 days.
Write bucket policies: Write a JSON bucket policy that: (a) allows your IAM user to read all objects, (b) allows an EC2 role to upload objects, and (c) denies all non-HTTPS requests.
SSE-KMS lab: Enable SSE-KMS on a bucket. Upload a file. Download it. Open CloudTrail and find the kms:GenerateDataKey and kms:Decrypt events corresponding to your upload and download.
Pre-signed URL: Using boto3, upload a private file to S3, generate a pre-signed URL valid for 5 minutes, open it in your browser, wait 5 minutes, try again — confirm it expires.
CLI practice: Using AWS CLI, sync a local folder to S3, list the objects, update one file locally, sync again, and confirm only the changed file was re-uploaded (not everything).
Versioning test: Enable versioning on a bucket. Upload a file, modify it, upload again with the same key, upload a third time. List all versions. Retrieve v1. Delete the latest version and confirm v2 becomes current.
Write the JSON and YAML: Without looking at notes, write from scratch: (a) a CloudFormation YAML snippet that creates an S3 bucket with versioning, (b) an S3 bucket policy in JSON that makes all objects in /public/* readable by everyone.

AWS Session 7 — S3 Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS ELB & Auto Scaling — Building Self-Healing, Scalable Infrastructure

Tejas Shinkar — Fri, 03 Jul 2026 01:51:46 +0000

Elastic Load Balancing & Auto Scaling

Traffic Distribution · ASG · Scaling Policies · ALB vs NLB · Health Checks · Zero-Downtime Deployments

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session covers the two services that make AWS infrastructure truly production-grade: ELB and Auto Scaling.

📋 Topics Covered

#	Topic	Type
1	Purpose of Load Balancing	Concept
2	How Load Balancing Works (Algorithms)	Concept + Interview
3	Horizontal vs Vertical Scaling	Concept + Interview
4	Scaling Policies — Dynamic, Scheduled, Predictive	Concept + Cert
5	Auto Scaling Group (ASG) — Core Config	Concept + Lab
6	ASG Lifecycle & Key Metrics	Concept + Cert
7	Health Checks & Replacement Policies	Concept + Lab
8	Session Stickiness	Concept + Interview
9	Connection Draining (Deregistration Delay)	Concept + Interview
10	Instance Refresh — Zero-Downtime Deployments	Concept + DevOps
11	Load Balancer Types — ALB vs NLB vs GWLB	Concept + Cert
12	Target Groups & Health Check Config	Concept + Lab
13	ALB Deep Dive — Routing Rules	Concept + Cert
14	Architecture — ALB + ASG	Concept + DevOps
15	Monitoring & Troubleshooting	Concept + DevOps
16	Best Practices	DevOps
17	AWS CLI Commands	Practical
18	Lab — Create ELB + Auto Scaling Group (7 Steps)	Lab
19	Cleanup Checklist	Lab
20	Assignment	Practice

Why Load Balancing Exists

Imagine a popular restaurant with one cashier — as soon as the lunch rush hits, the queue grows, the cashier gets overwhelmed, and eventually everything grinds to a halt. The fix? More cashiers, with a manager at the door directing customers to whoever is free.

A Load Balancer is that manager.

It sits in front of your EC2 instances and distributes incoming traffic across them — so no single server gets overwhelmed, and if one fails, the others keep running.

Without a Load Balancer:

Single point of failure — one server down means your app is down
No horizontal scaling — you're stuck with one server
No health checking — dead servers still receive traffic
Manual failover — someone has to intervene

With a Load Balancer:

High availability across multiple AZs
Horizontal scalability — add more servers on demand
Automatic health checks — dead servers removed instantly
Automatic failover — zero manual intervention
SSL/TLS termination — offloads HTTPS decryption from backend servers

How Load Balancing Works

Request flow — what actually happens:

1. User's browser sends HTTP request to your domain
2. DNS resolves your domain → returns ALB's IP address
3. Request hits the Load Balancer
4. LB picks a healthy backend EC2 instance
5. Forwards request to that instance
6. Instance processes and responds
7. LB returns the response to the user

Load Distribution Algorithms:

Algorithm	How it works	AWS Usage
Round Robin	Requests go in rotation: 1→2→3→1→2→3	ALB default
Least Connections	Route to instance with fewest active connections	Good for variable request lengths
IP Hash	Client IP always maps to same backend	Sticky sessions
Weighted	Instance A gets 70%, B gets 30%	Canary deployments
Flow Hash	Hash of protocol+IP+port	NLB default for TCP/UDP

🎯 Cert Tip: ALB uses weighted round robin by default. NLB uses flow hash for TCP/UDP traffic.

Horizontal vs Vertical Scaling

This is one of the most fundamental concepts in cloud architecture — and a guaranteed interview question.

Analogy: Your app is a delivery service. Vertical scaling = buying a bigger truck. Horizontal scaling = buying more trucks.

Vertical Scaling (Scale Up)

Upgrade the existing instance — stop it, change the type, restart.

t3.micro → t3.medium → t3.large → t3.xlarge

✅ Simple — no distributed system complexity
✅ Better for databases (data consistency is easier)
❌ Has a hard ceiling — biggest instance type is the limit
❌ Requires downtime during the type change
❌ Cannot auto-scale

Horizontal Scaling (Scale Out)

Add more instances — distribute load across them.

1 × t3.micro → 2 × t3.micro → 4 × t3.micro → back to 2

✅ Theoretically unlimited scalability
✅ Fault tolerant — one fails, others keep running
✅ Scales dynamically with Auto Scaling Groups
✅ No downtime when scaling
❌ Requires a Load Balancer to distribute traffic
❌ Requires stateless app design (or sticky sessions for state)

🎯 Cloud-native best practice: always use horizontal scaling with ASG. Vertical scaling is for legacy systems or databases that can't easily distribute state.

Scaling Policies — Three Types

How does AWS know when to scale? Scaling Policies define the trigger.

1. Dynamic Scaling — Target Tracking (Most Common)

Monitor a metric and automatically adjust to keep it at a target value.

Metric: Average CPU Utilization
Target: 50%

CPU rises to 75% → ASG adds instances until CPU ≈ 50%
CPU drops to 20% → ASG removes instances until CPU ≈ 50%

AWS creates two CloudWatch alarms automatically — one for scale-out, one for scale-in. You don't configure alarms manually.

Best for: Most production workloads. Simplest to set up and maintain.

2. Scheduled Scaling

Scale at pre-defined times — for predictable, known traffic patterns.

Every weekday 8:30 AM  → set Desired = 6  (office hours ramp up)
Every weekday 7:00 PM  → set Desired = 2  (evening wind down)
Every Saturday 12:00 AM → set Desired = 8 (weekend traffic spike)

Best for: Business-hours traffic, batch jobs at fixed times, known seasonal spikes.

3. Predictive Scaling (ML-based)

AWS analyses at least 2 weeks of historical metric data, forecasts future demand, and scales EC2 instances before traffic arrives — not in reaction to it.

Historical pattern: Every Friday at 5PM, traffic doubles
Predictive: ASG pre-scales at 4:45PM — instances ready before surge hits

Best for: Recurring, predictable patterns where reactive scaling causes latency spikes.

🎯 Cert Tip: Predictive Scaling requires a minimum of 2 weeks of historical data to build its forecast model.

Auto Scaling Group (ASG)

An ASG is the component that actually manages your EC2 fleet — launching, terminating, and replacing instances automatically.

Analogy: ASG is like a staffing agency for your servers. You tell it: "I need at least 2 people, ideally 4, maximum 10 — and if anyone quits or gets sick, hire a replacement immediately."

The Three Capacity Numbers (Core Concept)

MAX     [4] ──── Hard ceiling. ASG will NEVER exceed this, even under extreme load
DESIRED [2] ──── ASG always tries to maintain exactly this many healthy instances
MIN     [1] ──── Hard floor. ASG will NEVER go below this, even at zero traffic

ASG constantly compares running instances against Desired capacity. If an instance fails a health check, it terminates it and launches a replacement — no human intervention needed.

ASG Configuration Components

Setting	What it does
Launch Template	Blueprint for every EC2 ASG launches — AMI, instance type, SG, User Data
Min / Desired / Max	Capacity boundaries
Availability Zones	Which AZs to spread instances across (always 2+)
Load Balancer / Target Group	Where to register launched instances
Health Check Type	EC2 (VM-level) or ELB (app-level) — always use ELB
Scaling Policies	Dynamic, scheduled, or predictive rules
Termination Policy	Which instances to remove during scale-in (default: oldest)

✅ Always use Launch Templates not Launch Configurations — templates support versioning, Instance Refresh, and newer EC2 features.

ASG Lifecycle — Instance States

Scale-out triggered
      ↓
  Pending ──── being launched by ASG
      ↓
  InService ── running, healthy, serving traffic ← normal operating state
      ↓
  Draining ─── marked for removal, existing connections completing
      ↓
  Terminated ─ deleted, no longer in ASG

  Unhealthy ── failed health check → immediately replaced with new instance

Health Checks & Replacement Policies

Two Types of Health Checks

EC2 Health Check (VM level)
Checks if the instance is running — system and instance status checks pass. Doesn't know if your application is actually responding.

ELB Health Check (App level)
The ALB sends a real HTTP request to your configured path (e.g., /health). If the app doesn't return HTTP 200, the instance is marked unhealthy.

✅ Always use ELB health checks for web applications. An instance can be "running" while Apache is crashed inside — EC2 checks miss this entirely. ELB checks catch it within 60 seconds.

Health Check Configuration

Setting	Default	What it means
Interval	30 seconds	How often checks are sent
Timeout	5 seconds	How long to wait for response
Healthy Threshold	2 consecutive successes	Instance declared healthy
Unhealthy Threshold	2 consecutive failures	Instance declared unhealthy
Grace Period	300 seconds	Time after launch before checks begin (allows app boot time)

Self-Healing Flow

Health check fails twice
      ↓
ALB marks instance "Unhealthy" → stops sending traffic to it
      ↓
ASG receives notification
      ↓
ASG terminates the unhealthy instance
      ↓
ASG launches a new replacement from Launch Template
      ↓
Replacement passes health checks → added to Target Group
      ↓
Total time: ~5-10 minutes, fully automated ✅

Session Stickiness (Sticky Sessions)

By default, ALB routes each request to any healthy instance via round robin — a user's second request might go to a different server than their first. For stateless apps that's fine. For stateful apps (shopping carts, auth sessions), the session exists only on one server.

Sticky Sessions fixes this: ALB sets a cookie in the browser that pins the user to one specific backend instance for the session duration.

User logs in → ALB routes to EC2-1 → sets cookie: "stick to EC2-1"
Next request → user sends cookie → ALB routes to EC2-1 again ✅
Session expires / cookie deleted → back to normal round robin

When to use: Shopping carts, user authentication, stateful apps with in-memory sessions.

Downsides:

Uneven load distribution
If the pinned instance fails, the session is lost anyway

🎯 Proper long-term fix: Externalize session state to ElastiCache (Redis) or DynamoDB so any instance can serve any user without needing stickiness.

Connection Draining (Deregistration Delay)

When an instance is removed (scale-in, replacement, deployment), what happens to users mid-request on that instance?

Without draining: requests are immediately dropped → users get errors.

Connection Draining:

Instance marked for removal
      ↓
ALB immediately stops sending NEW requests to this instance
      ↓
ALB waits for EXISTING in-flight requests to finish
      ↓
Default wait: up to 300 seconds
      ↓
After all connections finish (or timeout) → instance terminated
      ↓
Zero dropped requests ✅

This is what enables zero-downtime scaling and deployments.

Instance Refresh — Zero-Downtime Deployments

When you update your application (new AMI, new config), you need to replace all running instances with the new version. Instance Refresh automates this within ASG.

Refresh strategies:

Strategy	Method	Speed	Risk
Rolling	One instance at a time	Slowest	Lowest
Rolling with Stagger	X% in parallel (25%→50%→75%)	Medium	Medium
Blue/Green	All new instances launched, traffic switched, old terminated	Fastest	Higher cost

Workflow:

Update Launch Template (new AMI version)
      ↓
Start Instance Refresh in ASG console
      ↓
ASG launches new instance from updated template
      ↓
Health checks pass on new instance
      ↓
Old instance: connection draining → terminated
      ↓
Repeat for all instances
      ↓
All instances running new AMI — zero downtime ✅

🎯 DevOps relevance: Instance Refresh is the ASG equivalent of a rolling update in Kubernetes. Same concept, different tool.

Load Balancer Types — ALB vs NLB vs GWLB

	ALB	NLB	GWLB
OSI Layer	Layer 7 (Application)	Layer 4 (Transport)	Layer 3 (Network)
Protocol	HTTP, HTTPS	TCP, UDP, TLS	All IP protocols
Routing basis	Path, host, header, query string	IP + Port (flow hash)	Transparent proxy
Latency	~25-100ms	<100 microseconds	Very low
Static IP	❌ No	✅ Yes	✅ Yes
SSL Termination	✅ Yes	✅ Yes	❌ No
Use Case	Web apps, APIs, microservices	Gaming, IoT, VoIP, real-time	Firewalls, IDS/IPS appliances

Decision guide:

Web app or REST API?                 → ALB (almost always)
Ultra-low latency TCP/UDP required?  → NLB
Traffic must pass through a firewall/security appliance? → GWLB

⚠️ Classic Load Balancer (CLB) exists but is legacy — never use for new deployments.

🎯 Cert scenarios: "Which LB supports path-based routing?" → ALB only. "Which provides a static IP?" → NLB. "Which for real-time gaming at massive scale?" → NLB.

ALB Deep Dive — Routing Rules

ALB operates at Layer 7 — it understands HTTP. This enables routing based on the actual content of each request.

Listener Rules (evaluated top-to-bottom):

Rule 1: Host = api.example.com
        → Forward to: api-target-group (3x t3.medium)

Rule 2: Host = admin.example.com
        → Forward to: admin-target-group (restricted access)

Rule 3: Path = /images/*
        → Forward to: media-target-group (or S3)

Rule 4: Path = /api/*
        → Forward to: api-target-group

Rule 5: HTTP (port 80) — any path
        → Redirect: 301 to HTTPS

Default Rule: no other rule matched
        → Forward to: web-target-group (main homepage)

One ALB can route to multiple backend services — this is the foundation of microservices architecture on AWS.

Target Groups & Health Check Config

A Target Group is a named pool of EC2 instances (or IPs, or Lambdas) that the ALB routes traffic to. One ALB → multiple target groups via listener rules.

Setting	Value
Target Type	Instances
Protocol	HTTP or HTTPS
Port	App's listening port (80, 8080, 3000, etc.)
VPC	Must match your EC2 instances
Health Check Path	`/health` — must return HTTP 200-299

⚠️ When using ASG — never manually register instances to the Target Group. Let ASG handle registration/deregistration automatically as instances launch and terminate.

Architecture — ALB + ASG (Production Standard)

                     Internet Users
                           │
                       Route 53
                  (domain → ALB DNS)
                           │
               Application Load Balancer
             (Internet-facing, 2+ AZs, SG-ALB)
                           │
                     Target Group
               (health-checked instance pool)
                           │
            Auto Scaling Group (min=1, desired=2, max=4)
         ┌─────────────────┼──────────────────┐
         │                 │                  │
    EC2 (AZ-a)         EC2 (AZ-b)        EC2 (AZ-b)
    SG-EC2, :80        SG-EC2, :80       SG-EC2, :80
         │
     (shared state)
    RDS Database
    or ElastiCache

CloudWatch CPU metric → Scaling Policy → ASG scales in/out

Three architectural pillars achieved:

High Availability — multi-AZ ALB and instances
Fault Tolerance — ASG self-heals on any instance failure
Cost Efficiency — scale down at night, up during peaks

Monitoring & Troubleshooting

Key CloudWatch Metrics

ALB Metrics:

Metric	Alert When
`TargetResponseTime`	> 500ms sustained
`RequestCount`	Sudden unexpected drop
`HTTPCode_Target_5XX`	> 0 sustained
`HealthyHostCount`	Below Desired capacity
`UnHealthyHostCount`	Greater than 0

ASG Metrics:

Metric	What it shows
`GroupDesiredCapacity`	Target count
`GroupInServiceInstances`	Healthy, running count
`GroupPendingInstances`	Currently launching
`GroupTerminatingInstances`	Being removed

Common Issues & Fixes

UnHealthyHostCount > 0

Check: SG-EC2 allows port 80 FROM SG-ALB (not from internet)
Check: Health check path actually returns 200 (try curl from EC2)
Check: App running → systemctl status httpd
Check: Grace period → did instance have 300s to boot before checks started?

ASG not scaling despite high CPU

Check: Scaling policy threshold — is CPU actually crossing it?
Check: Max capacity — already at the ceiling?
Check: Cooldown — previous scale action still in cooldown?
Check: CloudWatch alarm — is it actually in ALARM state?

Requests timing out after ALB

Check: EC2 instances responding? Try curl from inside the VPC
Check: Security group — SG-EC2 must allow HTTP FROM SG-ALB only
Check: App logs — any errors being thrown?

Session lost when instance replaced

Short-term: Enable sticky sessions on the Target Group
Long-term: Move session state to ElastiCache (Redis) or DynamoDB

Best Practices

ALB
  ✅ Always deploy across 2+ AZs
  ✅ Use path-based routing to consolidate microservices on one ALB
  ✅ Enable access logs for debugging and compliance
  ✅ Add HTTPS listener + redirect HTTP → HTTPS (use ACM for free cert)
  ✅ Use dedicated health check endpoint /health (not just /)

AUTO SCALING
  ✅ Use Target Tracking (simplest) over Step Scaling
  ✅ Set cooldown 300-600 seconds to avoid scaling oscillation
  ✅ Set Grace Period long enough for your app to boot (300s minimum)
  ✅ Use Launch Templates (not Launch Configurations)
  ✅ Test scaling with a load test before going to production
  ✅ Combine Scheduled + Target Tracking for predictable + reactive scaling

HEALTH CHECKS
  ✅ Always use ELB health checks (not EC2 only)
  ✅ Healthy threshold: 2, Unhealthy threshold: 2 (fast detection)
  ✅ Enable connection draining (default 300s)

SECURITY
  ✅ SG-EC2 inbound: HTTP from SG-ALB only (never 0.0.0.0/0)
  ✅ Users hit ALB only — EC2 instances never exposed to internet directly

AWS CLI Commands

# Create Application Load Balancer
aws elbv2 create-load-balancer \
  --name web-app-alb \
  --subnets subnet-aaa subnet-bbb \
  --security-groups sg-alb-xxx \
  --scheme internet-facing \
  --type application

# Create Target Group
aws elbv2 create-target-group \
  --name web-server-tg \
  --protocol HTTP --port 80 \
  --vpc-id vpc-xxx \
  --health-check-path /

# Create Auto Scaling Group
aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name web-app-asg \
  --launch-template LaunchTemplateName=web-server-lt,Version='$Latest' \
  --min-size 1 --max-size 4 --desired-capacity 2 \
  --availability-zones us-east-1a us-east-1b

# Attach Target Group to ASG
aws autoscaling attach-load-balancer-target-groups \
  --auto-scaling-group-name web-app-asg \
  --target-group-arns arn:aws:elasticloadbalancing:...

# Create Target Tracking Policy (CPU 50%)
aws autoscaling put-scaling-policy \
  --auto-scaling-group-name web-app-asg \
  --policy-name cpu-target-tracking \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration \
  '{"TargetValue":50.0,"PredefinedMetricSpecification":{"PredefinedMetricType":"ASGAverageCPUUtilization"}}'

# Describe ASG
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names web-app-asg

# Manually set desired capacity
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name web-app-asg \
  --desired-capacity 3

🧪 LAB — Create ELB + Auto Scaling Group

Security Groups (Create First)

SG-ALB:

Inbound:  HTTP (80) from 0.0.0.0/0
Outbound: All traffic

SG-EC2:

Inbound:  HTTP (80) from SG-ALB  ← reference the SG, not CIDR
Outbound: All traffic

⚠️ SG-EC2's inbound rule must reference SG-ALB as source — not 0.0.0.0/0. This blocks direct internet access to EC2. Only ALB can reach them.

Step 1 — Create Launch Template

EC2 → Launch Templates → Create launch template

Name:            web-server-lt
AMI:             Amazon Linux 2023 (64-bit x86)
Instance Type:   t2.micro / t3.micro (Free Tier)
Key Pair:        your-key-pair
Security Group:  SG-EC2
Storage:         8 GiB gp3

User Data (paste in Advanced Details):

#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/availability-zone)
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/placement/region)

cat > /var/www/html/index.html << EOF
<!DOCTYPE html>
<html>
<body style="font-family:Arial;text-align:center;padding:60px;background:#0A1628;color:#fff;">
  <h1 style="color:#FF9900;">Hello from AWS!</h1>
  <p>Instance ID: <b style="color:#FF9900;">$INSTANCE_ID</b></p>
  <p>Availability Zone: <b style="color:#FF9900;">$AZ</b></p>
  <p>Region: <b style="color:#FF9900;">$REGION</b></p>
</body>
</html>
EOF

✅ Checkpoint: web-server-lt listed in EC2 → Launch Templates

Step 2 — Create Target Group

EC2 → Load Balancing → Target Groups → Create target group

Target type:         Instances
Name:                web-server-tg
Protocol / Port:     HTTP / 80
VPC:                 Default VPC
Health check path:   /
Healthy threshold:   2
Unhealthy threshold: 2
Timeout:             5 seconds
Interval:            30 seconds

⚠️ On "Register targets" page — skip it entirely, click Create directly. ASG will register instances automatically.

✅ Checkpoint: web-server-tg shows status "unused" — this is correct, no instances yet.

Step 3 — Create Application Load Balancer

EC2 → Load Balancers → Create Load Balancer → Application Load Balancer

Name:           web-app-alb
Scheme:         Internet-facing
VPC:            Default VPC
Subnets:        Select 2 subnets in DIFFERENT AZs (mandatory — fails with one)
Security Group: SG-ALB (remove default SG)
Listener:       HTTP : 80 → Forward to web-server-tg

Wait ~2-3 minutes for state to become Active.

✅ Checkpoint: Copy ALB DNS name — you'll need it to test.

Step 4 — Create Auto Scaling Group

EC2 → Auto Scaling Groups → Create Auto Scaling group

Name:             web-app-asg
Launch template:  web-server-lt (Latest)
VPC + Subnets:    Same 2 subnets as ALB

Load balancing:   Attach to existing → web-server-tg | HTTP
Health check:     ELB, Grace period: 300 seconds

Desired: 2  |  Min: 1  |  Max: 4
Scaling: None for now (add in Step 5)

✅ Checkpoint: 2 EC2 instances launch within 1-2 min, both show Healthy in Target Group.

Step 5 — Configure Scaling Policy

ASG → web-app-asg → Automatic scaling → Create dynamic scaling policy

Policy type:   Target tracking scaling
Policy name:   cpu-target-tracking
Metric:        Average CPU Utilization
Target value:  50
Warmup:        300 seconds

AWS auto-creates two CloudWatch alarms (scale-out + scale-in).

Step 6 — Test & Verify

# Open ALB DNS in browser
http://<ALB-DNS-name>
# → "Hello from AWS!" + Instance ID + AZ

# Refresh 5-10 times
# → Instance ID changes = ALB routing to different instances ✅
# → AZ changes = multi-AZ distribution ✅

# Trigger scale-out (SSH into one EC2)
ssh -i your-key.pem ec2-user@<ec2-public-ip>
sudo yum install -y stress
stress --cpu 4 --timeout 600

# Watch in console:
# EC2 → Auto Scaling Groups → Activity tab (new instances launching)
# CloudWatch → Alarms (CPU alarm goes to ALARM state)

Expected timeline after stress starts:

T+0:00  CPU spikes
T+1:00  CloudWatch alarm: OK → ALARM
T+3:00  New EC2 launches
T+5:00  New instance healthy → ALB routes to 3 instances
T+10:00 Stress ends → CPU drops
T+18:00 ASG scale-in → back to Desired=2

Step 7 — Monitor with CloudWatch

CloudWatch → Metrics → EC2 → By Auto Scaling Group
  → web-app-asg → CPUUtilization → Period: 1 min

CloudWatch → Metrics → ApplicationELB
  → Add: RequestCount, HealthyHostCount, TargetResponseTime

🧹 Cleanup — Delete in This Exact Order

1. Auto Scaling Group  → terminates all EC2 instances automatically
   Wait 2-3 min for all instances to fully terminate

2. Application Load Balancer

3. Target Group

4. Launch Template

5. Security Groups  → delete SG-EC2 first, then SG-ALB
   (SG-ALB referenced by SG-EC2 — must delete SG-EC2 first)

6. CloudWatch Alarms  → delete auto-created scaling alarms

💰 ALB = ~$16/month + EC2 costs. Always clean up after labs.

📝 Assignment

Task 1 (Mandatory) — Prove Self-Healing:
Manually terminate one running EC2 instance. Watch ASG Activity tab — confirm it auto-launches a replacement. Verify ALB kept serving traffic with zero downtime.

Task 2 (Intermediate) — Step Scaling Policy:
Create a Step Scaling policy: add 2 instances if CPU > 70%, add 1 if CPU 50-70%, remove 1 if CPU < 20%.

Task 3 (Intermediate) — HTTPS:
Add HTTPS listener (port 443) to ALB. Use AWS Certificate Manager for a free certificate. Add listener rule: HTTP → HTTPS redirect.

Task 4 (Advanced) — Scheduled Scaling:
Scale up to 4 instances at 9AM weekdays, down to 1 at 6PM weekdays. Set times in UTC.

Task 5 (Advanced) — CloudWatch Dashboard:
Build a dashboard with 4 widgets: CPU%, RequestCount, HealthyHostCount, GroupInServiceInstances.

⚡ Quick Revision

LOAD BALANCING
  Manager directing traffic to healthy servers
  Eliminates single point of failure
  ALB algorithm: weighted round robin (default)

SCALING
  Horizontal = more instances (cloud-native, preferred)
  Vertical   = bigger instance (simple, has ceiling, needs downtime)

SCALING POLICIES
  Target Tracking = keep metric at target, AWS manages alarms (use this)
  Scheduled       = scale at specific times
  Predictive      = ML-based, needs 2 weeks history

ASG CAPACITY
  MIN     = hard floor, never below
  DESIRED = ASG always works to maintain this count
  MAX     = hard ceiling, never exceed

HEALTH CHECKS
  EC2  = is instance running? (basic)
  ELB  = is app returning HTTP 200? (always use this)
  Grace Period = time after launch before first check (300s default)

LB TYPES
  ALB  = Layer 7, HTTP/HTTPS, path/host routing → web apps
  NLB  = Layer 4, TCP/UDP, microsecond latency → gaming/IoT
  GWLB = Layer 3, security appliances, firewalls

STICKY SESSIONS
  Cookie pins user to one EC2 for session duration
  Proper fix: external session store (Redis/ElastiCache)

CONNECTION DRAINING
  In-flight requests complete before instance terminates
  Default: 300 seconds | Enables zero-downtime scaling

INSTANCE REFRESH
  Rolling AMI replacement across entire ASG fleet
  Zero-downtime | Kubernetes rolling update equivalent

SECURITY GROUP PATTERN
  SG-ALB: HTTP 80 from 0.0.0.0/0
  SG-EC2: HTTP 80 FROM SG-ALB only
  Direct EC2 access blocked from internet

💼 Interview Questions

Q1: Horizontal vs Vertical scaling — what's the difference and which does AWS recommend?
Horizontal = add more instances, requires LB, unlimited scale, fault tolerant. Vertical = upgrade existing instance, simple, has a hard ceiling, requires downtime. AWS recommends horizontal scaling for cloud-native apps.

Q2: What are Min, Desired, and Max capacity in an ASG?
Min = hard floor, ASG never drops below. Max = hard ceiling, ASG never exceeds. Desired = the target count ASG constantly works to maintain. Scaling policies adjust Desired within Min-Max boundaries.

Q3: Why use ELB health checks instead of EC2 health checks?
EC2 checks only verify the VM is running. ELB checks send a real HTTP request and verify the app returns 200. An EC2 can be "running" while Apache is crashed — EC2 checks miss it, ELB checks catch it within 60 seconds.

Q4: What is Connection Draining?
It allows in-flight requests to complete before an instance is terminated. ALB immediately stops sending new requests but waits up to 300 seconds for active connections to finish — enabling zero-downtime scaling and deployments.

Q5: ALB vs NLB — when do you use each?
ALB for HTTP/HTTPS web apps and microservices — understands application content, supports path/host routing. NLB for ultra-low latency TCP/UDP workloads (gaming, IoT, VoIP) or when a static IP is required.

Q6: Why reference SG-ALB as source in SG-EC2 instead of 0.0.0.0/0?
Referencing SG-ALB as source means only traffic that came through the ALB can reach EC2. Direct internet access to the EC2 IP is blocked. More secure, no IP management needed as instances scale.

Q7: What is Instance Refresh?
ASG's native mechanism to replace all running instances with a new Launch Template version (new AMI or config). It handles draining, replacement, and health checks automatically. Equivalent to a Kubernetes rolling update.

AWS Session 6 — ELB & Auto Scaling | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS EFS Essentials — Shared File Storage Across Multiple EC2 Instances

Tejas Shinkar — Wed, 01 Jul 2026 06:41:40 +0000

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session covers Amazon EFS — shared storage that multiple EC2 instances can read/write simultaneously.

📋 Topics Covered

#	Topic	Type
1	What is Amazon EFS	Concept
2	EFS vs EBS vs S3	Concept + Interview
3	EFS Architecture (Mount Targets, AZs, NFS)	Concept + Cert
4	Mounting — What It Means	Concept
5	Mount Targets & Security Group Config	Concept + Lab
6	EFS Storage Classes	Concept + Cert
7	EFS Lifecycle Management & Policies	Concept + Cert
8	EFS Performance Modes & Throughput Modes	Concept + Cert
9	Benefits of EFS	Concept
10	Security & Encryption	Concept + Interview
11	EFS Pricing & Cost Optimization	Concept
12	Lab: Launch 2 EC2 Instances	Lab
13	Lab: Create & Configure EFS	Lab
14	Lab: Mount EFS on Both Instances	Lab
15	Lab: Demonstrate File Sharing	Lab
16	Cleanup Checklist	Lab
17	Assignment — Independent Repeat	Practice

What is Amazon EFS?

EFS = Elastic File System — a fully managed, auto-scaling shared file system that multiple EC2 instances can mount and use at the same time, both reading and writing.

Analogy: Think of EBS as a personal hard drive — it belongs to one laptop only. EFS is like a shared Google Drive folder that your entire team can open simultaneously from different computers, see the same files, and edit them in real time.

Key characteristics:

Uses the NFS 4.1 protocol (Network File System — standard Linux file sharing)
Serverless — no capacity to provision, no servers to manage
Auto-scales — grows from KB to PB automatically, shrinks when files are deleted
Highly available — data replicated across multiple AZs in a Region
Pay-as-you-go — billed per GB actually stored, no pre-provisioning

EFS vs EBS vs S3 — The Big Comparison

This is one of the most commonly tested comparisons in AWS certifications.

Feature	EBS	EFS	S3
Access	Single EC2 instance only	Multiple EC2 instances simultaneously	Internet / API, unlimited clients
Protocol	Block storage	NFS v4.1	HTTP/REST (object storage)
Scope	AZ-specific	Regional (multi-AZ)	Region-wide, globally accessible
Scaling	Manual (resize volume)	Automatic	Unlimited, automatic
Performance	Very high, low latency	High (shared)	Variable
Use Case	OS disks, databases	Shared content, CMS, ML training	Backups, static assets, objects
Cost Model	Provisioned (pay for allocated size)	Usage-based (pay for what's stored)	Usage-based

Simple decision rule:

Need a disk for ONE EC2 instance (like an OS drive)?        → EBS
Need a shared folder MULTIPLE EC2 instances can access?      → EFS
Need to store files accessible from anywhere via internet?   → S3

🎯 Interview Tip: "Can two EC2 instances share an EBS volume?" — Only with io1/io2 Multi-Attach, and even then both must be in the same AZ. EFS is the natural answer when the question is about shared access across multiple instances in different AZs.

EFS Architecture

EFS spans multiple Availability Zones within a Region. Here's how it's structured:

                    Amazon EFS File System
                  (stored redundantly across AZs)
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                      │
      AZ 1                  AZ 2                   AZ 3
   Mount Target          Mount Target           Mount Target
   (ENI + IP)            (ENI + IP)             (ENI + IP)
        │                     │                      │
   EC2 Instance          EC2 Instance           EC2 Instance
   /mnt/efs              /mnt/efs               /mnt/efs

   All instances access the SAME file system via NFS (port 2049)

Mount Target — this is the key piece. It's an ENI (Elastic Network Interface) that EFS creates inside each AZ of your VPC. It's the "doorway" through which EC2 instances in that AZ connect to EFS over NFS.

One mount target per AZ (auto-created when you set up EFS)
Has its own private IP and security group
EC2 instances in that AZ connect to the local mount target for best performance

Mounting — What It Actually Means

Mounting is the process of attaching a file system to a directory on your OS so it behaves like a normal local folder.

Analogy: Mounting EFS is like plugging in a USB drive and having it appear as a folder you can open, save files into, and read from — except this "USB drive" lives in AWS and multiple computers can plug into it at once.

# Before mounting: /mnt/efs is just an empty folder
# After mounting: /mnt/efs IS the EFS file system
sudo mount -t nfs4 -o nfsvers=4.1 <EFS-DNS-Name>:/ /mnt/efs

Once mounted, every ls, cat, echo >, cp command on /mnt/efs is actually reading/writing to the shared EFS file system — not local disk.

Mount Targets & Security Group Configuration

This is the part that trips people up in labs — two separate security groups working together.

The Two Security Groups

EC2 Security Group (lab-ec2-sg)

Inbound:  SSH (22) from My IP
Outbound: NFS (2049) to EFS-SG  [allows EC2 to reach EFS]

EFS Security Group (lab-efs-sg)

Inbound:  NFS (2049) from lab-ec2-sg  [only EC2 instances can reach EFS]
Outbound: Allow all (default)

Why two security groups referencing each other?

This is a common AWS pattern — instead of allowing an IP range, you allow traffic from a specific security group. This means: "Only instances that have the EC2 security group attached are allowed to talk to EFS on port 2049." Clean, scalable, and secure — you never have to know or update IP addresses.

EC2 Instance (has lab-ec2-sg)
        │
        │  NFS traffic, port 2049
        ▼
EFS Mount Target (has lab-efs-sg, allows inbound from lab-ec2-sg)
        │
        ▼
Access Granted ✅

⚠️ Common lab mistake: Creating the EFS file system and trying to mount immediately — it fails because the EFS security group doesn't yet allow NFS traffic from the EC2 security group. You must add that inbound rule first.

Mount Target Requirements

One mount target per AZ in your VPC
Requires a subnet assignment
Has a private IP auto-assigned
DNS-based access — EFS gives you one DNS name that automatically resolves to the correct local mount target for whichever AZ you're connecting from

Persistence Across Reboots

By default, if your EC2 instance restarts, the EFS mount is gone — you'd have to mount it manually again. To make it automatic:

# Add to /etc/fstab
<EFS-DNS-Name>:/ /mnt/efs nfs4 defaults,_netdev 0 0

The _netdev option tells the OS to wait for networking to be ready before attempting the mount — critical since EFS is a network resource, not local disk.

EFS Storage Classes

Storage Class	Use Case	Cost
Standard	Active data, accessed frequently	Higher per GB
Infrequent Access (IA)	Files not accessed often	~50% cheaper per GB + a per-access fee

How it works: Standard is the default — best for files you're actively using. IA is for files that just sit there most of the time (old logs, archived reports) — cheaper to store, but costs a small fee each time you access them. The trade-off only makes sense if access is genuinely rare.

EFS Lifecycle Management & Policies

This is automatic tiering — EFS moves files between Standard and IA based on actual usage, without you lifting a finger.

File created → Standard storage (full price)
        │
        │  Not accessed for X days (your chosen threshold)
        ▼
Auto-moved to Infrequent Access (cheaper)
        │
        │  Someone accesses the file again
        ▼
Auto-moved back to Standard

Configurable transition periods: 7, 14, 30, 60, or 90 days of inactivity (or disabled entirely).

Why this matters: You don't have to manually audit and move files — EFS handles cost optimization automatically based on real access patterns. Set it once during file system creation and forget about it.

🎯 Cert Perspective: "How do you reduce EFS storage cost for rarely accessed files without manual intervention?" → Enable Lifecycle Management with a transition policy to IA.

EFS Performance Modes & Throughput Modes

These are two separate settings that often get confused — performance mode is about latency/concurrency, throughput mode is about data transfer speed.

Performance Modes (set once, cannot change later)

Mode	Optimized For	Use Case
General Purpose (default)	Lowest latency	Web serving, CMS, most workloads — use this unless you have a specific reason not to
Max I/O	Maximum throughput, more concurrent operations	Big data, media processing, highly parallel workloads

Throughput Modes (can change later)

Mode	How it Works	Use Case
Bursting (default)	Throughput scales with file system size, bursts higher when needed	Default choice, cost-effective for most workloads
Provisioned	You specify a guaranteed throughput, independent of storage size	Predictable high-throughput needs regardless of how much is stored
Max I/O	Optimized for highest throughput and ops/second	Heavy parallel processing

🎯 Cert Perspective: Know that Performance Mode is locked at creation, but Throughput Mode can be changed afterward. A scenario like "small file system but needs guaranteed high throughput" points to Provisioned Throughput.

Benefits of EFS

Benefit	What It Means
Scalability	Grows from KB to PB automatically, zero downtime
High Availability	Data redundantly stored across multiple AZs
Durability	Multiple copies protect against component failure
Simple Integration	Standard Linux NFS interface — no app code changes needed
Cost-Optimized	IA storage class cuts costs ~50% for inactive files

Security & Encryption

Network Security:

Security groups control all access (as covered above)
VPC isolation — EFS only accessible within your VPC by default
NFS traffic always on port 2049

Data Encryption:

At rest — via AWS KMS, enable when creating the file system
In transit — via TLS, add -o tls to your mount command

Access Control (3 layers):

File System Policy — IAM-style resource policy on the EFS itself (grant/deny access)
POSIX file permissions — standard Linux permissions (chmod, chown) work exactly as expected
IAM integration — controls who can call EFS API operations (create, delete, modify)

Best practices:

Always enable encryption when creating the file system (can't add it retroactively)
Restrict security groups to required ports only — never open NFS to 0.0.0.0/0
Use VPC security groups for isolation, exactly like the EC2-SG ↔ EFS-SG pattern above

EFS Pricing & Cost Optimization

What you pay for:

Standard Storage — per GB stored
Infrequent Access — ~50% lower per GB, plus a small fee each time data is retrieved

Cost optimization checklist:

✅ Enable Lifecycle Management — auto-move inactive files to IA
✅ Choose sensible transition days based on actual access patterns
✅ Use Bursting Throughput (default) unless you specifically need Provisioned
✅ Monitor usage via CloudWatch metrics
✅ Clean up unused files regularly

🧪 LAB — Full Hands-On Walkthrough

Lab Architecture

            VPC (Default)
        ┌─────────────────────────┐
        │   Availability Zone      │
        │  ┌──────────┐  ┌──────────┐
        │  │   EC2-1   │  │   EC2-2   │
        │  │ /mnt/efs  │  │ /mnt/efs  │
        │  └─────┬────┘  └─────┬────┘
        │        │  NFS (2049)  │
        │        └──────┬───────┘
        │                ▼
        │         Amazon EFS
        │       (Mount Target)
        └─────────────────────────┘

Security Groups:
  EC2-SG  → allows SSH (22) inbound from My IP
  EFS-SG  → allows NFS (2049) inbound from EC2-SG only

Lab Step 1 — Launch 2 EC2 Instances

Repeat this entire process twice (EC2-Instance-1 and EC2-Instance-2):

1. EC2 Console → Launch Instances
2. Name: EC2-Instance-1 (then EC2-Instance-2)
3. AMI: Amazon Linux 2 (HVM) — Free tier eligible
4. Instance type: t2.micro / t3.micro (Free tier)
5. Key Pair: Create new → lab-key-pair → .pem format
6. Network: Default VPC, any subnet, Auto-assign public IP: Enable
7. Security Group: Create lab-ec2-sg → Allow SSH (22) from My IP
8. Storage: 8 GB gp2 root volume (default)
9. Launch Instance — repeat for Instance 2

⚠️ Use the SAME key pair, VPC, and subnet for both instances — keeps the lab simple and ensures they can communicate.

Lab Step 2 — Create and Configure EFS

1. AWS Console → Services → EFS → Create file system
2. Name: lab-efs
3. VPC: Select SAME VPC as your EC2 instances
4. Quick Create (defaults) or Customize
5. EFS auto-creates mount targets in each AZ of the VPC
6. Edit mount target Security Group → create/use lab-efs-sg
7. Add Inbound Rule to lab-efs-sg: NFS (2049) from lab-ec2-sg
8. Create — note the File System ID: fs-xxxxxxxxx

EFS Settings used (defaults):

Performance mode:  General Purpose
Throughput mode:   Bursting
Encryption:        Enabled (KMS)
Storage class:     Standard (multi-AZ redundancy)

Security Group Rule Summary:

Security Group	Direction	Port	Source
lab-ec2-sg	Inbound	22 (SSH)	My IP
lab-efs-sg	Inbound	2049 (NFS)	lab-ec2-sg

Lab Step 3 — Mount EFS on Both Instances

Open two separate SSH terminals — one per instance. Run identical steps on both.

# Step 1: Update packages
sudo yum update -y

# Step 2: Install NFS/EFS utilities
sudo yum install -y amazon-efs-utils
# (or: sudo yum install -y nfs-utils — depending on AMI)

# Step 3: Create mount directory
sudo mkdir efs
# or: sudo mkdir /mnt/efs

# Step 4: Mount EFS
sudo mount -t efs -o tls fs-012f61809a9d6e846:/ /efs
# or using DNS name:
sudo mount -t nfs4 -o nfsvers=4.1 <EFS-DNS-Name>:/ /mnt/efs

# Step 5: Verify the mount
df -h
# or
mount | grep efs

⚠️ If the mount fails on the first attempt — this is expected if you haven't yet added the NFS inbound rule to the EFS security group. Add it, then retry the mount command. This is intentional in the lab flow to show why the security group rule matters.

Lab Step 4 — Demonstrate File Sharing (The Core Proof)

On EC2 Instance 1 (Writer):

cd /mnt/efs

# Write a file
echo 'Hello from EC2 Instance 1!' | sudo tee shared-file.txt

# Append a timestamp
echo "Timestamp: $(date)" | sudo tee -a shared-file.txt

# Create a folder and copy a file into it
sudo mkdir /mnt/efs/shared-folder
sudo cp /etc/hostname /mnt/efs/shared-folder/host1.txt

On EC2 Instance 2 (Reader) — same EFS mount:

cd /mnt/efs

# List files — see what Instance 1 wrote, instantly
ls -la /mnt/efs
# Output: shared-file.txt  shared-folder/

# Read the file Instance 1 created
cat shared-file.txt
# Output:
# Hello from EC2 Instance 1!
# Timestamp: Mon Jan 01 10:00:00 UTC

# Write back from Instance 2
echo 'Reply from Instance 2!' | sudo tee -a shared-file.txt

Verify on EITHER instance:

cat /mnt/efs/shared-file.txt
# Hello from EC2 Instance 1!
# Timestamp: Mon Jan 01 10:00:00 UTC
# Reply from Instance 2!

What this proves:

Both instances read/write the same file system simultaneously
Changes from one instance are instantly visible on the other
No manual file transfer, sync, or copying needed — this is the entire value of EFS

Permissions Note

# Grant full access to all users on the mount
sudo chmod 777 /mnt/efs

# Now you can write without sudo
echo 'No sudo needed!' > /mnt/efs/test.txt
cat /mnt/efs/test.txt

🧹 Cleanup Checklist (Don't Skip This!)

✅ Unmount EFS on both instances: sudo umount /mnt/efs
✅ Terminate both EC2 instances
✅ Delete EFS file system (lab-efs)
✅ Delete security groups: lab-ec2-sg, lab-efs-sg
✅ Delete key pair if no longer needed

⚠️ EFS bills per GB stored — even small test files left running add up over time if forgotten. Always clean up after labs.

📝 Assignment — Independent Repeat

Complete this without step-by-step guidance, using everything from the lab above:

1.  Launch EC2 #1 — Amazon Linux 2, t2.micro, new key pair, public IP enabled
2.  Launch EC2 #2 — same config, same key pair, same VPC/subnet
3.  Create EFS — name: my-efs, default VPC, default settings
4.  Configure SGs — EC2 SG: SSH (22). EFS SG: NFS (2049) from EC2 SG
5.  Mount on EC2 #1 — install nfs-utils, mkdir /mnt/efs, mount EFS
6.  Mount on EC2 #2 — same mount steps
7.  Verify Mounts — run df -h on both, confirm EFS appears
8.  Write from EC2 #1 — create a file with your name at /mnt/efs/yourname.txt
9.  Read from EC2 #2 — cat /mnt/efs/yourname.txt, confirm visible
10. Write from EC2 #2 — append a second line to the same file
11. Cleanup — unmount, terminate instances, delete EFS and SGs

Submission deliverables (screenshots):

EC2 Instances list — both Running
EFS file system page — showing created file system
df -h output on Instance 1 — /mnt/efs mounted
df -h output on Instance 2 — /mnt/efs mounted
Terminal — file written on Instance 1
Terminal — same file read from Instance 2

Grading breakdown: EC2 launch (25 pts) · EFS + SG config (25 pts) · Mounted on both (25 pts) · File sharing both ways (25 pts)

⚡ Quick Revision

WHAT IS EFS
  Fully managed, shared, auto-scaling NFS file system
  Multiple EC2 instances mount it simultaneously
  Protocol: NFS v4.1, Port: 2049

EFS vs EBS vs S3
  EBS = single instance, block storage, AZ-locked
  EFS = multiple instances, NFS, Region-wide (multi-AZ)
  S3  = internet/API access, object storage, unlimited

ARCHITECTURE
  Mount Target = ENI in each AZ, the doorway to EFS
  One per AZ, auto-created, has its own private IP + SG

SECURITY GROUP PATTERN
  EC2-SG  → outbound NFS to EFS-SG
  EFS-SG  → inbound NFS (2049) FROM EC2-SG only
  Never open NFS to 0.0.0.0/0

STORAGE CLASSES
  Standard = frequent access, higher cost
  IA       = infrequent access, ~50% cheaper + access fee

LIFECYCLE POLICY
  Auto-moves files Standard → IA after X days inactive (7/14/30/60/90)
  Auto-moves back to Standard when accessed again

PERFORMANCE MODES (set once)
  General Purpose = default, lowest latency, most workloads
  Max I/O = max throughput, more concurrency, big data

THROUGHPUT MODES (changeable)
  Bursting     = default, scales with FS size
  Provisioned  = fixed guaranteed throughput
  Max I/O      = highest throughput/ops

ENCRYPTION
  At rest  → KMS (enable at creation)
  In transit → TLS (-o tls in mount command)

MOUNT COMMAND
  sudo mount -t efs -o tls fs-xxxx:/ /efs
  or
  sudo mount -t nfs4 -o nfsvers=4.1 <EFS-DNS>:/ /mnt/efs

PERSISTENCE
  Add to /etc/fstab with _netdev option to survive reboot

💼 Interview Questions

Q1: What is the main difference between EFS and EBS?
EBS is block storage attached to a single EC2 instance within one AZ — like a personal hard drive. EFS is a shared NFS file system that multiple EC2 instances across multiple AZs can mount and access simultaneously — like a shared network drive.

Q2: Why does an EFS mount initially fail in the lab even though the file system was created successfully?
The EFS security group doesn't yet have an inbound rule allowing NFS (port 2049) traffic from the EC2 security group. Until that rule is added, EC2 instances cannot reach the mount target, so the mount command times out or fails.

Q3: How does EFS achieve high availability?
EFS automatically creates a mount target in each Availability Zone of your VPC and stores data redundantly across multiple AZs within the Region — so it tolerates an AZ failure without data loss or downtime.

Q4: What's the difference between EFS Performance Mode and Throughput Mode?
Performance Mode (General Purpose or Max I/O) is set once at creation and determines latency characteristics — it can't be changed later. Throughput Mode (Bursting, Provisioned, or Max I/O) controls how much data transfer capacity you get and can be changed after creation.

Q5: A company has files that are rarely accessed after 30 days but must remain available. How would you optimize EFS costs?
Enable EFS Lifecycle Management with a transition policy set to 30 days. Files inactive for 30 days automatically move to Infrequent Access storage class at roughly 50% lower storage cost. If accessed again, they automatically move back to Standard.

Q6: Why do EC2-SG and EFS-SG reference each other instead of using IP addresses?
Referencing security groups instead of IPs is more secure and scalable — it means "only resources carrying this specific security group can connect," regardless of how many instances exist or what their IPs are. No manual IP management needed as you scale.

🔬 Practice Tasks

Complete the Assignment above independently (full EC2 + EFS lab without referring to lab steps).
After mounting EFS on one instance, deliberately remove the NFS inbound rule from the EFS security group, then try writing a new file. Observe what happens. Re-add the rule and confirm it works again.
Enable Lifecycle Management on your EFS with a 7-day transition to IA. Create a file, wait (or simulate), and check the EFS Console to see which storage class it reports.
Add your EFS mount to /etc/fstab on one instance. Reboot the instance and confirm the mount persists automatically without running the mount command again.
Compare: create a 1GB file directly on EBS root volume vs on the EFS mount. Time how long a cp operation takes on each, and note any latency differences.

AWS Session 5 — EFS Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

AWS EC2 Essentials — Everything You Need to Know (With Real-Life Analogies)

Tejas Shinkar — Mon, 29 Jun 2026 08:47:58 +0000

EC2 Essentials

Compute · Storage · Networking · Security · Lifecycle · CLI

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. Every session gets documented with hands-on steps, analogies, and interview prep.

📋 Topics Covered

#	Topic	Type
1	What is EC2 & Why It Exists	Concept
2	EC2 Instance Types & Families	Concept + Cert
3	Elastic Network Interface (ENI)	Concept + Interview
4	Amazon EBS — Persistent Storage	Concept + Lab
5	EBS Volume Types & IOPS	Concept + Cert
6	EBS Snapshots	Concept + Lab
7	EBS Multi-Attach & Encryption	Concept + Cert
8	EBS vs S3	Concept + Interview
9	Instance Store — Temporary Storage	Concept + Cert
10	AMI — Amazon Machine Image	Concept + Lab
11	EC2 Key Pairs	Concept + Lab
12	Security Groups	Concept + Interview
13	Elastic IP	Concept + Lab
14	EC2 Instance Lifecycle	Concept + Cert
15	EC2 Status Checks	Concept + Cert
16	EC2 Placement Groups	Concept + Cert
17	EC2 Billing & Pricing Models	Concept + Cert
18	AWS CLI for EC2	Practical
19	boto3 — Python SDK for EC2	Practical
20	Lab Recap	Lab

Why EC2 Matters

Before cloud, running an application meant buying a physical server — weeks of procurement, rack installation, OS setup, then maintenance forever. EC2 changed that entirely.

EC2 = Elastic Compute Cloud — a virtual server running on AWS hardware that you can provision in minutes, pay for by the hour, and terminate when done. No physical hardware, no upfront cost, no idle servers gathering dust.

The name breaks down simply:

E — Elastic → scale up or down on demand
C — Compute → processing power for your workloads
C — Cloud → delivered over the internet, not from your office

EC2 Instance Types — Choosing the Right Size

Every EC2 instance has a type — a predefined combination of CPU, RAM, and network capacity. AWS groups them into families, each optimized for a different kind of workload.

Analogy: Think of instance types like vehicles. You don't drive a truck to pick up groceries, and you don't use a hatchback to transport 10 tonnes of cargo. Right tool for the right job.

Family	Examples	Optimized For	Real Use Case
General Purpose	t3, m5	Balanced CPU + RAM	Web servers, small databases
Compute Optimized	c5, c6	High CPU	Batch processing, ML inference
Memory Optimized	r5, x1	Large RAM	In-memory databases, Redis
Storage Optimized	i3, h1	Fast local disk	NoSQL databases, data warehousing
Accelerated Computing	p3, g4	GPU	Machine learning training, video rendering

Instance naming: t3.medium

t = family (general purpose, burstable)
3 = generation (newer = better price/performance)
medium = size (nano → micro → small → medium → large → xlarge → 2xlarge...)

Free Tier: t2.micro or t3.micro — 750 hours/month free for 12 months. Use this for all labs.

🎯 Cert Perspective: SAA-C03 frequently tests which instance type fits a given scenario. Know the families and their use cases cold.

AWS Compute Optimizer — if you already have EC2 running and want to know if you're over/under-provisioned, this ML-based service analyses your actual usage and recommends the right instance type. Useful in production, not needed for labs.

ENI — Your EC2's Network Card

ENI = Elastic Network Interface — the virtual network card that gives your EC2 instance its network identity.

Analogy: ENI is like a SIM card in a phone. The phone (EC2) needs the SIM (ENI) to have a phone number (IP address), connect to a network, and send/receive data.

Every EC2 instance gets a primary ENI automatically at launch. You can attach additional secondary ENIs for advanced use cases.

What an ENI carries:

Attribute	What it is
Private IPv4	Internal IP within your VPC (always present)
Public IPv4	Internet-facing IP (assigned if enabled, changes on restart)
Elastic IP	Static public IP you allocate manually (doesn't change)
MAC Address	Unique network hardware identifier
Security Group	Firewall rules applied to this network interface

Why multiple ENIs?

Separate management traffic from application traffic
Move a network identity (with its Elastic IP and security groups) from one instance to another during failover — without changing DNS or IP

🎯 Interview Tip: "How do you move an Elastic IP from a failed EC2 to a healthy one?" — Detach the ENI (or reassociate the Elastic IP) from the old instance and attach to the new one. Zero IP change for clients.

EBS — Persistent Storage for EC2

EBS = Elastic Block Store — think of it as the hard disk for your EC2 instance. It's where your OS, application files, and data live.

Analogy: EC2 is your laptop. EBS is the hard drive inside it. If you sell the laptop (terminate EC2), you can remove the hard drive first (detach EBS) and plug it into a new laptop (attach to new EC2). The data stays.

Key Characteristics

Persistent — data survives EC2 stop/restart (unlike Instance Store)
Attached over network — not physically inside the EC2 host, but connected via high-speed AWS network
AZ-locked — an EBS volume lives in one specific AZ. You cannot directly attach it to an EC2 in a different AZ
Replicated within AZ — AWS automatically replicates your EBS data within the same AZ for durability

EBS Volume Types

Type	IOPS	Throughput	Best For
gp3 (General Purpose SSD)	3,000–16,000	1,000 MB/s	Most workloads, default choice
io2 (Provisioned IOPS SSD)	Up to 64,000	1,000 MB/s	High-performance databases
st1 (Throughput Optimized HDD)	N/A	Up to 500 MB/s	Big data, log processing
sc1 (Cold HDD)	N/A	Up to 250 MB/s	Infrequent access, archival

IOPS = Input/Output Operations Per Second — how many read/write operations storage can handle per second. Higher IOPS = faster storage = more expensive.

Analogy: IOPS is like a highway's capacity. gp3 is a 4-lane road — good for normal traffic. io2 is a 10-lane expressway — needed when hundreds of vehicles (database queries) rush through simultaneously.

🎯 Cert Perspective: Know gp3 as the default choice and io2 for databases requiring high IOPS. SAA-C03 gives scenarios and asks which volume type fits.

EBS Snapshots

A Snapshot is a point-in-time backup of your EBS volume, stored in S3.

Incremental — first snapshot copies everything, subsequent ones only copy what changed (saves cost and time)
Cross-AZ/Region — snapshots can be restored as new EBS volumes in any AZ or Region
Create AMIs — you can create a custom AMI from a snapshot
Taken while running — no need to stop your instance

When to use snapshots:

Before making risky changes to your server
Creating a backup before terminating an instance
Copying data to another Region or AZ

EBS Multi-Attach

Normally, one EBS volume = one EC2 instance. Multi-Attach breaks this rule.

Only available on io2 volumes
Attach one volume to up to 16 EC2 instances simultaneously in the same AZ
Use case: clustered databases that need shared storage
Requires cluster-aware file systems — not a standard use case for beginners

🎯 Interview Tip: "Can two EC2 instances share the same EBS volume?" — Yes, but only io2 with Multi-Attach, and only within the same AZ.

EBS Encryption

Encrypts data at rest (stored on disk) and in transit (moving between EC2 and EBS)
Uses AWS KMS (Key Management Service) for key management
Zero performance impact
Can be enabled by default at account level

To use EBS data in another AZ:

EBS Volume (AZ-a) → Create Snapshot → Copy Snapshot to AZ-b
→ Restore Snapshot as new EBS Volume (AZ-b) → Attach to EC2 in AZ-b

EBS vs S3 — Common Confusion Cleared

	EBS	S3
Type	Block storage	Object storage
Access	Attached to EC2 like a disk	Via API/URL from anywhere
Use For	OS, databases, application files	Images, videos, backups, static files
Persistence	Survives stop, lost on terminate (if not enabled)	Permanent until you delete
Region	AZ-specific	Region-wide, globally accessible
Analogy	Hard drive inside your laptop	Google Drive in the cloud

Instance Store — Fast but Temporary

Instance Store is physical storage built directly into the EC2 host machine — not a network-attached disk like EBS.

Analogy: Instance Store is like RAM in your laptop — incredibly fast, but everything is wiped when you turn it off.

	Instance Store	EBS
Speed	Very high (physically attached)	High (network-attached)
Persistence	❌ Lost on stop/terminate/host failure	✅ Persists
Cost	Included with instance	Charged separately
Use Case	Temporary files, cache, buffers	OS, databases, permanent data

⚠️ Never store anything important on Instance Store. If the underlying host fails, data is gone with zero recovery.

AMI — The Blueprint for EC2

AMI = Amazon Machine Image — a pre-configured template that contains everything needed to launch an EC2 instance: OS, software, configurations, and the root volume snapshot.

Analogy: AMI is like a cookie cutter. Once you have the perfect cookie shape (your configured server), you stamp out identical copies as many times as you want, instantly.

AMI Sources

Source	What it is
AWS Provided	Amazon Linux 2, Ubuntu, Windows — official, maintained by AWS
AWS Marketplace	Pre-built AMIs with commercial software (e.g. WordPress, Nginx)
Community AMIs	Shared by other AWS users — use with caution
Custom AMI	You build your own from a configured EC2 instance

EBS-Backed vs Instance Store-Backed

	EBS-Backed	Instance Store-Backed
Can stop instance?	✅ Yes	❌ No — only terminate
Data on root volume	Persists after termination (if enabled)	Lost on termination
Performance	Good	Very high
Use for	Production, anything important	Stateless, ephemeral workloads

Almost always use EBS-backed AMIs. Instance store-backed is a legacy option.

AMI is Region-Specific

AMIs do not automatically replicate across Regions. If you want to use the same AMI in Mumbai and Singapore, you must copy the AMI to the target Region manually.

EC2 Console → AMIs → Select AMI → Actions → Copy AMI → Choose Region

🎯 Cert Perspective: "You built a custom AMI in us-east-1 and want to launch the same in ap-south-1." Answer: Copy the AMI to ap-south-1 first, then launch from there.

Creating a Custom AMI

Launch EC2 → Install software + configure everything
→ EC2 Console → Select Instance → Actions
→ Image and Templates → Create Image
→ Name it → Create
→ New AMI appears in AMIs section
→ Use it to launch identical instances anytime

Key Pairs — Your SSH Access

A Key Pair is a public-private cryptographic pair used to log into EC2 without a password.

Analogy: A padlock and key. AWS puts the padlock (public key) on your EC2. You hold the key (private key / .pem file). Only someone with the key can open it.

How it works:

You create Key Pair in AWS Console
  ↓
AWS stores Public Key on EC2 (in ~/.ssh/authorized_keys)
  ↓
You download Private Key (.pem file) — ONCE, never again
  ↓
To connect: ssh -i my-key.pem ec2-user@<public-ip>
  ↓
EC2 verifies: "Does this private key match my public key?" → Yes → Access granted

Access methods by OS:

Linux/Mac: SSH with .pem file
Windows: RDP (Remote Desktop Protocol) for Windows instances
Browser: EC2 Instance Connect — no key needed, direct from Console

⚠️ Critical: If you lose the .pem file, you permanently lose SSH access. There is no "forgot key" option. Store it safely. Never share it.

Security Groups — The Firewall

A Security Group is a stateful virtual firewall controlling what traffic can reach your EC2 instance.

Analogy: Security Group is the bouncer at a club. You give the bouncer a list (rules) — only people on the list get in. Everyone else is turned away by default.

Key behaviours:

Default: deny all inbound, allow all outbound
Stateful — if you allow inbound traffic on port 80, the response automatically goes out even without an explicit outbound rule
Multiple EC2 instances can share one security group
One EC2 instance can have multiple security groups

Common rules:

Purpose	Protocol	Port	Source
SSH access	TCP	22	Your IP only
Web server (HTTP)	TCP	80	0.0.0.0/0 (everyone)
Web server (HTTPS)	TCP	443	0.0.0.0/0 (everyone)
Windows RDP	TCP	3389	Your IP only
MySQL	TCP	3306	App server's security group

⚠️ Never open port 22 to 0.0.0.0/0 in production. Restrict SSH to your specific IP only. Bots scan for open port 22 constantly.

🎯 Interview Tip: "What's the difference between a Security Group and a NACL?" — Security Group is stateful (tracks connections), operates at instance level. NACL is stateless (needs explicit inbound + outbound rules), operates at subnet level. Security Groups are your first answer for EC2 traffic control.

Elastic IP — Static Public Address

By default, when you stop and restart an EC2 instance, its public IP changes. This breaks DNS records, bookmarks, and API integrations.

Elastic IP solves this — it's a static public IPv4 address that stays the same regardless of instance stop/start.

Analogy: A regular EC2 public IP is like a temporary hotel room number — changes every visit. Elastic IP is like having a permanent postal address that forwards to wherever you're staying.

Important pricing rule:

✅ Free when associated with a running EC2 instance
❌ Charged (~$0.005/hr) when allocated but NOT associated (sitting idle)

⚠️ Always release Elastic IPs when done with labs. An unattached Elastic IP silently charges you. This is one of the most common sources of unexpected AWS bills for beginners.

Lab steps (what you did):

Allocate Elastic IP:
  EC2 Console → Elastic IPs → Allocate Elastic IP → Allocate

Associate with EC2:
  Select Elastic IP → Actions → Associate
  → Choose Instance → Associate

Disassociate when done:
  Select Elastic IP → Actions → Disassociate

Release (to stop charges):
  Select Elastic IP → Actions → Release Elastic IP Address

EC2 Instance Lifecycle

Every EC2 instance goes through states — understanding these determines what you're charged for.

Launch
  ↓
Pending (starting up, not billed yet)
  ↓
Running (fully operational, billed per second)
  ↓          ↓                ↓
Stopping   Rebooting      Terminating
  ↓          ↓                ↓
Stopped    Running        Terminated
(EBS billed, (brief        (deleted forever,
 compute not) interruption)  not recoverable)

State	Compute Charge	EBS Charge	Recoverable?
Running	✅ Yes	✅ Yes	—
Stopped	❌ No	✅ Yes	✅ Yes — start again
Terminated	❌ No	❌ No (deleted)	❌ No — gone forever
Hibernated	❌ No	✅ Yes	✅ Yes — resumes from where it left off

Stop vs Terminate — know the difference:

Stop = shut down, like turning off your laptop. EBS data stays. You can start it again.
Terminate = delete permanently. Everything gone (unless you enabled "Delete on Termination = No" for EBS).

⚠️ Lab habit: Always stop instances after practice. Terminate only when fully done with the lab and you've confirmed nothing important is on the volume.

EC2 Status Checks

AWS automatically runs health checks on every EC2 instance every minute.

Two types:

System Status Check — checks AWS infrastructure (power, network, hardware host)

If this fails: the problem is on AWS's side
Fix: Stop and Start the instance (moves it to a new physical host)

Instance Status Check — checks the OS and software inside your EC2

If this fails: the problem is inside your instance (OS crash, bad config)
Fix: Reboot the instance or SSH in and fix the issue

🎯 Cert Perspective: "System status check fails — what do you do?" → Stop and Start (NOT reboot — reboot keeps it on the same host). This is a classic exam question.

CloudWatch Integration: You can set alarms on status checks to trigger automatic recovery actions — this is how production systems self-heal without human intervention.

EC2 Placement Groups

When you launch multiple EC2 instances, AWS decides where to place them on physical hardware. Placement Groups let you control this for specific needs.

Three strategies:

Cluster — all instances packed tightly in one AZ, on hardware close together

Ultra-low latency between instances, high network throughput
Risk: if the rack fails, all instances are affected
Use for: HPC (High Performance Computing), big data jobs that need fast inter-node communication

Spread — instances placed on separate physical racks, even separate data centers within AZ

Maximum fault isolation — rack failure only affects one instance
Limit: max 7 instances per AZ per placement group
Use for: small groups of critical instances that absolutely cannot fail together

Partition — instances spread across partitions (groups of racks), multiple instances per partition

Balance between performance and fault isolation
Use for: Hadoop, Kafka, Cassandra — distributed systems with many nodes

Analogy:

Cluster = teammates sitting at the same table (fast to talk, but one fire affects all)
Spread = teammates in different buildings (isolated, but limited seats)
Partition = teammates in different floors of the same building (grouped but isolated from other groups)

🎯 Cert Perspective: Scenario-based questions. "Low-latency HPC cluster" → Cluster. "7 critical instances must never fail together" → Spread. "Hadoop with 200 nodes" → Partition.

EC2 Billing — What You Actually Pay For

EC2 pricing has multiple components — not just the instance:

Component	What's Charged
Instance Hours	Per second (minimum 60 seconds) based on type + region
EBS Storage	Per GB-month
Snapshots	Per GB-month stored in S3
Data Transfer OUT	Charged for traffic leaving AWS to internet
Elastic IP (idle)	Charged when not attached to running instance

Three purchasing models:

Model	Commitment	Savings	Use For
On-Demand	None	0%	Learning, unpredictable workloads
Reserved Instances	1-3 years	30-70%	Steady, predictable production workloads
Spot Instances	None (can be interrupted)	Up to 90%	Batch jobs, fault-tolerant workloads

AWS Pricing Calculator: Before any deployment, estimate costs at calculator.aws. Select service → region → configuration → get monthly estimate. Good habit to build before every project.

AWS CLI — Managing EC2 from Terminal

AWS CLI lets you control AWS from your terminal — no Console clicking needed. Essential for DevOps automation.

Installation:

# Linux/Mac
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

Configuration:

aws configure
# Enter:
# AWS Access Key ID
# AWS Secret Access Key
# Default region: ap-south-1
# Output format: json

Credentials are stored in ~/.aws/credentials and ~/.aws/config.

⚠️ Never hardcode credentials in scripts or push to GitHub. Use IAM Roles with Instance Profiles for EC2 (as covered in Session 3).

Common EC2 CLI commands:

# List all instances
aws ec2 describe-instances

# Launch a new instance
aws ec2 run-instances --image-id ami-0abcdef1234567890 \
  --instance-type t2.micro --key-name my-key

# Stop an instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Start an instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0

# Terminate an instance
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0

# List security groups
aws ec2 describe-security-groups

boto3 — Controlling EC2 with Python

boto3 is the AWS SDK for Python — lets you write scripts that interact with AWS services programmatically.

pip install boto3

import boto3

# List all EC2 instances and their states
ec2 = boto3.resource('ec2')
for instance in ec2.instances.all():
    print(f"ID: {instance.id} | Type: {instance.instance_type} | State: {instance.state['Name']}")

# Stop a specific instance
instance = ec2.Instance('i-1234567890abcdef0')
instance.stop()
print("Instance stopped")

# Start a specific instance
instance.start()
print("Instance started")

🎯 DevOps Relevance: You'll use boto3 to automate EC2 management — auto-start/stop instances on a schedule, build custom monitoring, automate AMI creation. This is where your Python-for-DevOps foundation directly connects to AWS.

Lab Recap — What You Practiced

✅ Launched EC2 instance (Amazon Linux 2, t2.micro)
✅ Added EBS volume during launch
✅ Allocated Elastic IP and attached to instance
✅ Verified public IP stays same after restart
✅ Terminated instance
✅ Removed EBS volumes
✅ Released Elastic IP (important — stops billing!)

⚡ Quick Revision

EC2 INSTANCE TYPES
  t-family = General Purpose (Free Tier: t2.micro)
  c-family = Compute Optimized
  r-family = Memory Optimized
  i-family = Storage Optimized
  p/g-family = GPU / Accelerated Computing

STORAGE
  EBS     = Persistent network disk (survives stop, AZ-locked)
  S3      = Object storage (internet-accessible, Region-wide)
  Instance Store = Temporary physical disk (wiped on stop/terminate)

EBS TYPES
  gp3  = Default choice, general workloads
  io2  = High-performance databases, Multi-Attach
  st1  = Big data throughput
  sc1  = Cold/infrequent data

AMI
  Blueprint for EC2 (OS + software + config)
  Region-specific → must copy to use in another Region
  EBS-backed = can stop/start ✅ (use this always)

NETWORKING
  ENI = Virtual network card (carries IPs, MAC, SGs)
  Elastic IP = Static public IP (free when attached, charged when idle!)
  Security Group = Stateful firewall (default: deny all in, allow all out)

LIFECYCLE
  Running → billed
  Stopped → EBS billed only, can restart
  Terminated → gone forever, nothing billed

STATUS CHECKS
  System check fails → Stop + Start (new host)
  Instance check fails → Reboot (fix OS)

PLACEMENT GROUPS
  Cluster   = Low latency, same AZ, HPC
  Spread    = Max isolation, separate racks, critical apps
  Partition = Distributed systems, many nodes

PRICING
  On-Demand  = Pay per second, no commitment
  Reserved   = 1-3 yr commitment, 30-70% off
  Spot       = Cheapest, can be interrupted

💼 Interview Questions

Q1: What is the difference between stopping and terminating an EC2 instance?
Stopping shuts the instance down — EBS data persists, compute billing stops, you can start it again. Terminating permanently deletes the instance and (by default) its root EBS volume. There is no recovery from termination.

Q2: An EC2 instance is in a different AZ than your EBS volume. Can you attach it?
No. EBS volumes are AZ-locked — they can only attach to EC2 instances in the same AZ. To use data across AZs, take a snapshot of the volume and restore it as a new volume in the target AZ.

Q3: What is the difference between Instance Store and EBS?
Instance Store is physical storage attached to the EC2 host — very fast but completely lost when the instance stops, terminates, or the host fails. EBS is persistent network-attached storage that survives stop/restart and can be detached and reattached to other instances.

Q4: A system status check is failing on your EC2. What do you do?
Stop and Start the instance (not reboot). This moves the instance to a new physical host in AWS infrastructure, resolving hardware-level issues. Rebooting keeps it on the same host and won't fix a system-level failure.

Q5: What is an Elastic IP and when would you use it?
An Elastic IP is a static public IPv4 address. You'd use it when your application needs a permanent, unchanging public IP — for example, when DNS records point to your server or when clients whitelist specific IPs. Note: it's charged when not attached to a running instance, so always release when not in use.

Q6: When would you use a Cluster vs Spread placement group?
Cluster for workloads needing ultra-low latency between instances (HPC, big data processing) — all instances are packed close together in one AZ. Spread for critical instances that must be isolated from each other's hardware failures — each instance goes on a separate physical rack.

Q7: What are the three EC2 pricing models and when would you use each?
On-Demand for learning and unpredictable workloads — pay per second, no commitment. Reserved Instances for steady production workloads that run 24/7 — 1-3 year commitment saves 30-70%. Spot Instances for fault-tolerant batch jobs or flexible workloads — cheapest option but AWS can interrupt with 2-minute notice.

🔬 Practice Tasks

Launch + User Data: Launch an EC2 instance with a User Data script that installs Apache and serves a custom HTML page. Access it via the public IP in your browser.
Elastic IP lab: Launch an EC2 instance. Note its public IP. Stop it — note the IP changed. Now allocate an Elastic IP, associate it, stop/start the instance — confirm the IP stayed the same. Release the Elastic IP when done.
EBS volume: Launch an EC2 instance. Attach a second EBS volume (not root). SSH into the instance, format the volume, mount it, create a file. Stop the instance and confirm the file persists when you restart.
Custom AMI: Configure an EC2 instance (install Nginx, create a webpage). Create a custom AMI from it. Launch a second EC2 from that AMI — confirm Nginx is already installed and running.
CLI practice: Using AWS CLI, list all running EC2 instances in your account. Stop one using its instance ID via CLI. Start it again via CLI. All without touching the Console.
boto3 script: Write a Python script using boto3 that lists all EC2 instances in ap-south-1 and prints their instance ID, type, and current state.
Pricing Calculator: Use calculator.aws to estimate monthly cost of: 1 × t3.micro EC2 (Linux, ap-south-1, 24/7) + 20 GB gp3 EBS + 100 GB data transfer out.

AWS Session 4 — EC2 Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

APIs with `requests`, HTTP Methods & Status Codes, API Authentication, Boto3 (AWS SDK) Intro

Tejas Shinkar — Sun, 21 Jun 2026 17:41:45 +0000

📌 Key Concepts Overview

Concept	One-Line Definition
API	A way for two applications/systems to communicate over HTTP
`requests`	Python library for making HTTP requests (GET, POST, etc.)
HTTP Method	The "verb" describing what action you want (GET, POST, PUT, PATCH, DELETE)
Status Code	A number the server returns indicating success/failure of a request
`response.json()`	Parses the response body as a Python dict/list
Authentication Token	A secret credential used to prove identity to an API
`boto3`	The official AWS SDK for Python — used to automate AWS resources
`boto3.client()`	Creates a low-level client to interact with a specific AWS service

🌐 Part 1 — What Is an API?

API = Application Programming Interface — a defined way for one piece of software to talk to another. In DevOps, you'll constantly interact with APIs: AWS APIs, GitHub APIs, Slack webhooks, monitoring tool APIs (Datadog, PagerDuty), CI/CD tool APIs (Jenkins, GitLab).

import requests

The requests library is the standard, most widely used Python tool for making HTTP API calls.

📬 Part 2 — HTTP Methods

Method	Purpose	DevOps Example
`GET`	Fetch/read data	Get list of EC2 instances, check API health
`POST`	Send/create new data	Create a new resource, trigger a deployment
`PUT`	Update FULL data (replace entirely)	Replace an entire config object
`PATCH`	Update PARTIAL data (only changed fields)	Update just one field in a config
`DELETE`	Remove data	Delete a resource, terminate an instance

# Basic GET request
import requests
URL = "https://www.python.org/"
response = requests.get(URL)
print(response)               # <Response [200]>
print(response.status_code)    # 200
print(response.text)            # raw HTML/text content of the response

🔢 Part 3 — HTTP Status Codes

Code	Meaning	Category
`200`	SUCCESS	2xx — Success
`201`	CREATED	2xx — Success
`400`	BAD REQUEST	4xx — Client Error
`401`	UNAUTHORIZED	4xx — Client Error (missing/invalid auth)
`403`	FORBIDDEN	4xx — Client Error (no permission)
`404`	NOT FOUND	4xx — Client Error
`415`	UNSUPPORTED MEDIA	4xx — Client Error
`500`	SERVER ERROR	5xx — Server Error

Quick mental model:

2xx = it worked
4xx = YOU (the client) did something wrong — bad input, missing auth, wrong URL
5xx = THE SERVER broke — not your fault

# Always check status_code before trusting the response
if response.status_code == 200:
    print('✅ Request succeeded')
elif response.status_code == 401:
    print('❌ Authentication failed — check your token')
elif response.status_code == 404:
    print('❌ Resource not found')
else:
    print(f'⚠️ Unexpected status: {response.status_code}')

📦 Part 4 — Working with the Response Object

# .json() — parses response body as a Python dict/list (when API returns JSON)
data = response.json()
print(data)

# .headers — metadata about the response (server, content-type, rate limits, etc.)
print(response.headers)
print(response.headers.get("Content-Type"))    # get a SPECIFIC header value

response.text vs response.json():

Property/Method	Returns	Use when
`response.text`	Raw string	The response is HTML or plain text
`response.json()`	Python dict/list	The response is JSON (most modern APIs)

🔐 Part 5 — API Authentication

Most real-world APIs require authentication — you can't just call them anonymously like python.org.

Using `getpass` to Securely Capture a Token

from requests.auth import HTTPBasicAuth
import getpass

url = "https://api.github.com/user"
username = "your_github_username"
my_token = getpass.getpass("Enter your github token: ")   # hides input — doesn't echo to screen

headers = {"Authorization": f"Bearer {my_token}"}
response = requests.get(url, headers=headers)
print(response.status_code)
print(response.json())

Why getpass.getpass() instead of input()? It doesn't display what you type on screen — critical for entering tokens/passwords so they're not visible to anyone looking at your screen or captured in a saved notebook output.

The Authorization Header Pattern

import requests

token = "your_github_personal_access_token"

r = requests.get(
    "https://api.github.com/user",
    headers={"Authorization": f"Bearer {token}"}
)

print(r.json())

⚠️ CRITICAL SECURITY WARNING:

The class notebook had a real-looking GitHub token hardcoded directly in the code (token = "ghp_..."). Never do this in real work or in anything you save/commit/share — including Dev.to posts or GitHub repos. Hardcoded tokens in notebooks or scripts are one of the most common ways credentials leak publicly. If a token like this is ever committed to GitHub or shared anywhere public, revoke it immediately in GitHub Settings → Developer Settings → Personal Access Tokens, and generate a new one.

Best practice instead:
import os
token = os.environ.get('GITHUB_TOKEN')     # read from environment variable
# or use getpass.getpass() for interactive scripts, as shown above

🕸️ Part 6 — Web Content Fetching + Regex Extraction

import urllib.request
from re import findall

url = "https://www.summet.com/dmsi/html/codesamples/addresses.html"
response = urllib.request.urlopen(url)
html = response.read()
print(html, type(html))      # raw bytes, <class 'bytes'>

print(html.decode())           # decode bytes → readable string

Extracting Phone Numbers with Regex (from class)

# Pattern to extract US-style phone numbers like (660) 663-4518
data = findall(r'\(\d{3}\)\s\d{3}-\d{4}', html.decode())
for i in data:
    print(i)

Working pattern is:

r'\(\d{3}\)\s\d{3}-\d{4}'
# \(        — literal opening parenthesis
# \d{3}     — exactly 3 digits (area code)
# \)        — literal closing parenthesis
# \s        — one whitespace character
# \d{3}     — exactly 3 digits
# -         — literal hyphen
# \d{4}     — exactly 4 digits

DevOps relevance: This combination — fetching a URL's content and extracting structured data with regex — is exactly how you'd scrape status pages, parse third-party documentation for config values, or pull structured info out of HTML-based dashboards that don't offer a clean JSON API.

☁️ Part 7 — Boto3 (AWS SDK for Python) — Introduction

Boto3 is the official Python library/SDK for automating AWS. This is the single most important library for your DevOps/Cloud career — almost all AWS automation in Python goes through Boto3.

import boto3
from pprint import pprint     # pretty-print — formats nested dicts/lists readably

iam_client = boto3.client('iam')      # create a "client" for the IAM service
iam_client.list_users()                  # returns a dict of all IAM users
pprint(iam_client.list_users())           # same data, but readable/formatted

The `boto3.client()` Pattern

# Syntax: boto3.client('service_name')
iam_client = boto3.client('iam')
ec2_client = boto3.client('ec2')      # (not in class yet, but same pattern)
s3_client = boto3.client('s3')         # (same pattern, different service)

This is the foundational pattern for ALL of Boto3: create a client for a service, then call methods on it that map to AWS API actions.

Looping Through API Response Data

import boto3
iam_client = boto3.client('iam')
users = iam_client.list_users()

for i in users['Users']:           # 'Users' key holds a LIST of user dicts
    print(i['UserName'])             # each user is a dict — access fields by key

This is a direct continuation of the dict/list knowledge from earlier classes — AWS API responses are nested Python dicts and lists, exactly like the structures practiced in Class 2 and Class 6.

Creating Resources (Preview — Incomplete in Class)

# Creating an IAM user (class left this incomplete — syntax shown for reference)
import boto3
iam_client = boto3.client('iam')
iam_client.create_user(UserName='new-devops-user')    # correct method name is create_user (singular)

Note: The notebook had iam_client.create_users() (plural) which is not a valid Boto3 method — the correct method is create_user() (singular), which requires at minimum a UserName parameter. This was likely a class typo/work-in-progress — worth double-checking against the Boto3 IAM docs when you actually use it.

☁️ DevOps / Cloud Use Cases

# 1. Health check script using requests
import requests

def check_service_health(url):
    try:
        response = requests.get(url, timeout=5)
        if response.status_code == 200:
            print(f'✅ {url} is healthy')
        else:
            print(f'⚠️ {url} returned {response.status_code}')
    except requests.exceptions.RequestException as e:
        print(f'❌ {url} unreachable: {e}')

check_service_health('https://api.github.com')

# 2. Trigger a CI/CD pipeline via API (POST pattern)
import requests
import os

token = os.environ.get('CI_TOKEN')
headers = {'Authorization': f'Bearer {token}'}
payload = {'branch': 'main', 'environment': 'staging'}

# response = requests.post('https://ci.example.com/api/trigger', headers=headers, json=payload)

# 3. List all running EC2 instances with Boto3 (preview of next classes)
import boto3
ec2_client = boto3.client('ec2')
# instances = ec2_client.describe_instances()
# for reservation in instances['Reservations']:
#     for instance in reservation['Instances']:
#         print(instance['InstanceId'], instance['State']['Name'])

# 4. Audit all IAM users (directly from class)
import boto3
iam_client = boto3.client('iam')
users = iam_client.list_users()
for user in users['Users']:
    print(f"User: {user['UserName']} | Created: {user['CreateDate']}")

# 5. Securely load a GitHub token from environment instead of hardcoding
import os
import requests

token = os.environ.get('GITHUB_TOKEN')      # ✅ never hardcode tokens
headers = {'Authorization': f'Bearer {token}'}
response = requests.get('https://api.github.com/user', headers=headers)
print(response.status_code)

❌ Common Mistakes

Mistake	Issue	Fix
Hardcoding tokens/secrets in code	Massive security risk — gets leaked via Git history, notebooks, screenshots	Use `os.environ.get()` or `getpass.getpass()`
Not checking `status_code` before using `.json()`	Calling `.json()` on a failed/non-JSON response → crash	Check `response.status_code == 200` first
Using `response.text` for JSON APIs	Gives raw string, not a usable dict	Use `response.json()` for JSON responses
`iam_client.create_users()` (plural)	Not a real Boto3 method	Correct method is `create_user()` (singular)
Forgetting `headers` dict for authenticated requests	Gets `401 Unauthorized`	Always pass `headers={"Authorization": f"Bearer {token}"}`
Typo'd regex with unescaped `\d`	`d\{4}` instead of `\d{4}` — silently matches wrong things or errors	Double-check regex syntax — `\d{4}` not `d\{4}`
Not setting a `timeout` on `requests.get()`	Script can hang forever on an unresponsive server	Always pass `timeout=5` (or similar)

🎯 Interview Points

"What's the difference between GET, POST, PUT, and PATCH?"
→ GET retrieves data without modifying it. POST creates a new resource. PUT replaces an entire resource. PATCH updates only specific fields of a resource.
"What does a 401 vs 403 status code mean?"
→ 401 Unauthorized means you're not authenticated at all (missing/invalid credentials). 403 Forbidden means you ARE authenticated, but don't have permission for that specific action.
"Why should you never hardcode API tokens in your code?"
→ Hardcoded secrets can be leaked through version control history, shared notebooks, or screenshots — and unlike a password, anyone who obtains the token can impersonate your account/service until it's revoked.
"What's the difference between response.text and response.json()?"
→ response.text returns the raw response body as a string. response.json() parses that body as JSON and returns a Python dict/list — but only works if the response is actually valid JSON.
"What is Boto3 and how does it work at a high level?"
→ Boto3 is AWS's official Python SDK. You create a client for a specific service (boto3.client('iam'), boto3.client('ec2')), then call methods on that client that correspond to AWS API actions — responses come back as Python dicts/lists.
"How would you securely pass a secret/token in a Python script?"
→ Use environment variables (os.environ.get('TOKEN')) for automated/scheduled scripts, or getpass.getpass() for interactive scripts — never hardcode it directly in source code.

📚 Knowledge Base — Quick Revision

# ── REQUESTS BASICS ─────────────────────────────────────────
import requests
response = requests.get(url)               # GET request
response = requests.post(url, json=data)    # POST request

response.status_code        # e.g. 200, 404, 500
response.text                 # raw string body
response.json()                # parsed dict/list (if JSON)
response.headers                # response metadata dict

# ── HTTP METHODS ────────────────────────────────────────────
GET     # fetch
POST    # create
PUT     # full update
PATCH   # partial update
DELETE  # remove

# ── STATUS CODES ────────────────────────────────────────────
200 OK | 201 Created | 400 Bad Request | 401 Unauthorized
403 Forbidden | 404 Not Found | 500 Server Error

# ── AUTHENTICATION ──────────────────────────────────────────
import getpass
token = getpass.getpass('Enter token: ')        # hidden input
headers = {"Authorization": f"Bearer {token}"}
response = requests.get(url, headers=headers)

# ✅ secure: read from environment
import os
token = os.environ.get('MY_TOKEN')

# ── BOTO3 (AWS SDK) ──────────────────────────────────────────
import boto3
client = boto3.client('service_name')    # e.g. 'iam', 'ec2', 's3'
client.list_users()                        # example IAM call
client.create_user(UserName='name')         # correct method (singular)

from pprint import pprint
pprint(response_dict)                          # readable nested output

🏋️ Practice Questions

Easy

Make a GET request to https://api.github.com (no auth needed) and print the status code and the Content-Type header.
What is the difference between a 404 and a 500 status code? Give a real-world example of each.
Use getpass.getpass() to securely capture a fake "password" from the user and print its length (not the value itself).

Medium

Write a function check_status(url) that makes a GET request, and based on the status code, prints "Healthy", "Not Found", "Unauthorized", or "Server Error" accordingly. Use try-except to catch connection failures too.
Make an authenticated GET request to https://api.github.com/user using a personal access token read from an environment variable (os.environ.get('GITHUB_TOKEN')) rather than hardcoded. Print just the login field from the JSON response.
Use boto3.client('iam') to list all IAM users in your AWS account, and print each username along with their CreateDate.

DevOps-Focused

Multi-Endpoint Health Checker: Write a script that takes a list of URLs (e.g., ['https://api.github.com', 'https://www.python.org']), loops through them, makes a GET request to each with a 5-second timeout, and prints a health summary table (URL → status code → Healthy/Unhealthy). Wrap each request in try-except so one failure doesn't stop the loop.
IAM User Auditor: Using boto3.client('iam'), write a script that lists all IAM users and flags (prints a warning for) any user whose CreateDate is more than 365 days ago — a common security audit task (old, possibly unused credentials).

Exception Handling (`else`/`finally`, `raise`, Debugging Tools) + Regular Expressions (`re` module)

Tejas Shinkar — Sun, 21 Jun 2026 17:37:37 +0000

📌 Key Concepts Overview

Concept	One-Line Definition
`try-except-else-finally`	The complete exception-handling block structure
`else` (in try block)	Runs ONLY if no exception occurred in `try`
`finally`	Always runs — exception or not — used for cleanup
`sys.exc_info()`	Returns details about the exception currently being handled
`traceback` module	Prints the full error traceback (where exactly it failed)
`raise`	Manually triggers an exception
Regex (`re` module)	Pattern matching/searching/extracting from text
Metacharacter	Special symbol with reserved meaning in regex (`.`, `^`, `$`, `[]`)
Quantifier	Controls how many times a pattern repeats (`*`, `+`, `?`, `{n,m}`)

🧯 Part 1 — Exception Handling: Complete Structure

Nested `try-except` (Two Risky Operations)

try:                                    # outer suspicious code
    x = int(input('Enter a number: '))
    y = int(input('Enter a number: '))
    try:                                  # inner suspicious code
        c = x/y
        print(c)
    except:
        print('Division by zero not possible')
except:
    print('Incorrect input. Expecting an integer number only')

This works, but nesting try blocks gets messy. Python gives a cleaner way: else.

`else` — Runs ONLY If `try` Succeeds (No Exception)

try:
    x = int(input('Enter a number: '))
    y = int(input('Enter a number: '))
except:
    print('Incorrect input. Expecting an integer number only')
else:                                     # only runs if try block had NO error
    try:
        c = x/y
        print(c)
    except:
        print('Division by zero not possible')

Why else is better than just stuffing more code in try: It clearly separates "code that might fail" (in try) from "code that should only run if everything succeeded" (in else). Cleaner intent, easier to read.

lst = [73, 0, 6, 17, 0, 38, 10, 0, 47, 27]

try:
    print(10/lst[1])           # lst[1] = 0 → ZeroDivisionError
except:
    print('Exception caught')
else:
    print('If there is no exception caught, then this block gets executed')
print('Python')
# Output: 'Exception caught' then 'Python' — else block SKIPPED because an error occurred

try:
    print(10/lst[0])            # lst[0] = 73 → no error
except:
    print('Exception caught')
else:
    print('If there is no exception caught, then this block gets executed')
print('Python')
# Output: 3.65..., then 'If there is no exception...', then 'Python' — else block RUNS

`finally` — Always Runs, No Matter What

lst = [73, 0, 6, 17, 0, 38, 10, 0, 47, 27]

try:
    print(10/lst[1])           # ZeroDivisionError
except:
    print('Exception caught')
else:
    print('If there is no exception caught, then this block gets executed')
finally:
    print('This block of code will always get executed')
# Output: Exception caught → This block of code will always get executed
# (else is SKIPPED because an error occurred, but finally ALWAYS runs)

The Full Picture:

Block	When it runs
`try`	Always attempted first
`except`	Only if `try` raises an error
`else`	Only if `try` did NOT raise an error
`finally`	ALWAYS — regardless of error or not

DevOps relevance for finally: Closing a network connection, releasing a file lock, or cleaning up a temp resource — things that MUST happen whether the operation succeeded or failed.

🔍 Part 2 — Debugging Tools for Exceptions

`sys.exc_info()` — Get Details About the Current Exception

import sys

lst = [73, 0, 6, 17, 0, 38, 10, 0, 47, 27]
try:
    print(10/lst[1])
except:
    print(sys.exc_info())          # returns (exception_type, exception_value, traceback_object)
else:
    print('If there is no exception caught, then this block gets executed')
finally:
    print('This block of code will always get executed')

# Access individual parts:
print(sys.exc_info()[0])    # the exception TYPE, e.g. <class 'ZeroDivisionError'>
print(sys.exc_info()[1])    # the exception MESSAGE, e.g. division by zero

`traceback` Module — Print Where Exactly It Failed

import traceback

lst = [73, 0, 6, 17, 0, 38, 10, 0, 47, 27]
try:
    print(10/0)
    print(lst[42])             # never reached — error happens above first
except:
    print(traceback.print_exc())     # prints the FULL traceback — file, line number, error type
else:
    print('If there is no exception caught, then this block gets executed')
finally:
    print('This block of code will always get executed')

sys.exc_info() vs traceback.print_exc():

Tool	Use case
`except Exception as e: print(e)`	Quick, readable error message
`sys.exc_info()`	Programmatic access to type/value/traceback object
`traceback.print_exc()`	Full human-readable traceback — best for debugging logs

# The cleanest, most common production pattern:
try:
    x = int(input('Enter a number: '))
    y = int(input('Enter a number: '))
    print(x/y)
except Exception as e:
    print(e)             # simple, clear, catches everything with a readable message

🚨 Part 3 — `raise` — Manually Triggering Exceptions

# raise keyword is used to raise a user-defined error
try:
    a = int(input('Enter your age: '))
    if a < 18:
        raise Exception                  # manually trigger an exception
    else:
        print('Driving license application accepted')
except:
    print('Age is less than 18. Not eligible for the DL')

Why use raise instead of just an if-else with print()?

raise lets you enforce business rules as actual ERRORS — which propagate up, can be caught at a higher level, logged consistently, or stop execution entirely
Useful in larger systems where a function needs to signal "something is wrong" to whoever called it, not just print a message and continue

# Compare:
# Without raise — just a print, program flow continues normally
if a < 18:
    print('Age is less than 18')      # this is just informational

# With raise — flow JUMPS to except, treated as a real error
if a < 18:
    raise Exception                    # forces the except block to run

DevOps relevance: Validating config values, environment variables, or API responses — raise lets you fail fast and loud instead of silently continuing with bad data.

# Practical DevOps example
def validate_port(port):
    if port < 1 or port > 65535:
        raise ValueError(f'Invalid port: {port}')
    return port

try:
    validate_port(99999)
except ValueError as e:
    print(e)

☁️ DevOps / Cloud Use Cases (Exception Handling)

# 1. Cleanup pattern with finally — close connections regardless of outcome
try:
    # connection = connect_to_db()
    print('Connected to DB')
    # risky_query()
except Exception as e:
    print(f'Query failed: {e}')
finally:
    print('Closing DB connection')      # ALWAYS runs — connection never left hanging

# 2. else for "only proceed if input was valid"
try:
    instance_count = int(input('How many instances to launch? '))
except ValueError:
    print('Must be a number')
else:
    if instance_count > 10:
        print('⚠️ Warning: launching more than 10 instances')
    print(f'Launching {instance_count} instances...')

# 3. raise for config validation — fail fast
def check_region(region):
    valid_regions = ['ap-south-1', 'us-east-1', 'eu-west-1']
    if region not in valid_regions:
        raise ValueError(f'{region} is not a supported region')
    return True

try:
    check_region('mars-central-1')
except ValueError as e:
    print(f'Config error: {e}')

# 4. traceback for production logging
import traceback
try:
    # deploy_to_server()
    1/0   # simulated failure
except Exception:
    error_log = traceback.format_exc()
    # write error_log to a log file for later debugging
    print('Deployment failed — error logged')

❌ Common Mistakes (Exception Handling)

Mistake	Issue	Fix
Putting everything in `try`, nothing in `else`	Mixes "risky" code with "safe, dependent" code	Use `else` for code that should only run on success
Forgetting `finally` for cleanup	Resources (files, connections) may leak on error	Use `finally` for guaranteed cleanup
Using `print()` instead of `raise` for real errors	Program continues with bad data silently	Use `raise` to stop and signal failure properly
Bare `except:` hides the real problem	Hard to debug — don't know WHAT failed	Use `except Exception as e: print(e)` minimum
Confusing `sys.exc_info()[0]` and `[1]`	`[0]` = type, `[1]` = value/message	Remember the order: (type, value, traceback)

🎯 Interview Points (Exception Handling)

"What's the purpose of the else block in try-except?"
→ It runs only when the try block completes WITHOUT any exception — useful for code that depends on the try block succeeding, keeping it separate from the risky code itself.
"What's the difference between else and finally?"
→ else runs only if NO exception occurred. finally runs ALWAYS — exception or not. finally is for guaranteed cleanup.
"What does raise do?"
→ Manually triggers an exception, which can be caught by an except block — used to enforce validation rules or signal errors explicitly rather than silently continuing.
"What does sys.exc_info() return?"
→ A tuple of (exception_type, exception_value, traceback_object) for the exception currently being handled.
"Why use traceback.print_exc() over a simple print(e)?"
→ It shows the FULL traceback — exact file and line number where the error occurred — much more useful for debugging in logs than just the error message.

📚 Knowledge Base — Quick Revision (Exception Handling)

# ── FULL TRY STRUCTURE ──────────────────────────────────────
try:
    risky_code()
except SpecificError:
    handle_specific()
except Exception as e:
    print(e)                  # catch-all with message
else:
    only_if_success()          # runs ONLY if no exception
finally:
    always_runs()                # runs NO MATTER WHAT

# ── DEBUGGING TOOLS ─────────────────────────────────────────
import sys
sys.exc_info()           # (type, value, traceback)
sys.exc_info()[0]         # exception type
sys.exc_info()[1]          # exception message/value

import traceback
traceback.print_exc()      # full traceback, prints directly
traceback.format_exc()      # full traceback, returns as string (for logging)

# ── RAISE ────────────────────────────────────────────────────
if condition:
    raise Exception           # generic
    raise ValueError('msg')    # specific, with message

🔎 Part 4 — Regular Expressions (`re` module)

What Is Regex?

A pattern-matching language used to find, validate, extract, or replace specific patterns in text. In Python, accessed via import re.

DevOps relevance: Parsing log files, validating IP addresses/emails/ports, extracting fields from unstructured text (logs, config files, API responses), sed/grep-style operations but in Python.

The 4 Components of Regex

1. Functions          — match(), search(), findall(), finditer()
2. Metacharacters      — . ^ $ [] | ()
3. Special Sequences   — \w \W \s \S \d \D \b \B
4. Quantifiers         — ? * + {n} {n,m}

🔧 Regex Functions

`re.match()` — Match ONLY at the Beginning of the String

import re

text = 'India is my country. I love India very much'
pattern = r'India'
res = re.match(pattern, text)
print(res)              # <re.Match object; span=(0, 5), match='India'>

pattern = r'In'
res = re.match(pattern, text)
print(res)               # matches — 'In' is still at the start

pattern = r'love'              # NOT at the beginning
res = re.match(pattern, text)
print(res)                      # None — match() ONLY checks position 0

Get the actual matched text:

res = re.match(r'India ', text)
print(res.group())         # 'India ' — the matched string itself

`re.search()` — Find the FIRST Occurrence ANYWHERE

text = 'India is my country. I love India very much'
pattern = r'India'
res = re.search(pattern, text)
print(res)                  # finds it — anywhere in the string, not just the start

text2 = 'india is my country. I love India very much'   # lowercase first 'india'
pattern = r'India'
res = re.search(pattern, text2)
print(res)                   # finds the SECOND 'India' (capital I) — case-sensitive!

Match Object Attributes

text = 'india is my country. I love India very much'
pattern = r'India'
res = re.search(pattern, text)

print(res.group())     # the matched text itself: 'India'
print(res.span())       # (start, end) tuple — e.g. (29, 34)
print(res.start())       # just the start index: 29
print(res.end())          # just the end index: 34 (exclusive)

`re.findall()` — ALL Occurrences as a LIST of Strings

text = 'India is my country. I love India very much'
pattern = r'India'
res = re.findall(pattern, text)
print(res, type(res))      # ['India', 'India'] <class 'list'>

pattern = r'india'          # lowercase — won't match capital 'India'
res = re.findall(pattern, text)
print(res)                   # [] — empty list, case-sensitive, no match

`re.finditer()` — ALL Occurrences as MATCH OBJECTS (Iterator)

text = 'We are learning Python. This is Python class. Python is very interesting.'
pattern = r'Python'
res = re.finditer(pattern, text)
for i in res:
    print(i)              # full match object
    print(i.group())       # just the matched text
    print(i.span())         # (start, end) for EACH occurrence

findall() vs finditer() — Key Difference:

Function	Returns	Gives you `span()`/`start()`/`end()`?
`findall()`	List of plain strings	❌ No — just the text
`finditer()`	Iterator of match objects	✅ Yes — full details per match

🔣 Metacharacters

`[]` — Character Set

text = 'chcwasj393e*&#^(@babeidbisadjeiesna'

re.finditer(r'a', text)        # matches literal 'a' wherever it appears
re.finditer(r'[a]', text)       # same as above, set with ONE character

re.finditer(r'[acw]', text)      # matches a OR c OR w — each one individually
re.finditer(r'[1234]', text)      # matches any digit 1, 2, 3, or 4

re.finditer(r'[a-z]', text)        # matches any lowercase letter a through z
re.finditer(r'[A-J]', text)         # matches any uppercase letter A through J
re.finditer(r'[0-9]', text)          # matches any digit
re.finditer(r'[A-Za-z0-9]', text)     # matches any alphanumeric character

re.finditer(r'[^K]', text)             # ^ INSIDE [] means NOT — matches everything EXCEPT 'K'
re.finditer(r'[^A-Z]', text)            # matches everything EXCEPT uppercase letters

⚠️ ^ means two DIFFERENT things depending on context:

Position	Meaning
`r'^India'` (outside `[]`)	Match at the BEGINNING of the string
`r'[^A]'` (inside `[]`)	Match anything EXCEPT 'A' (negation)

`^` — Anchor to Start of String

text = 'India is my country. I love India'
pattern = r'^India'
res = re.finditer(pattern, text)
for i in res:
    print(i.group(), i.span())
# Only matches the FIRST 'India' — because it's at position 0

`$` — Anchor to End of String

text = 'This is Python class. Today is Saturday'
pattern = r'Saturday$'
res = re.finditer(pattern, text)
for i in res:
    print(i.group(), i.span())
# Matches only if 'Saturday' is the VERY LAST thing in the string

text2 = 'This is Python class. We have class on Saturday and Sunday. Today is Saturday'
pattern = r'Saturday$'
# Only matches the LAST 'Saturday' (the one at the true end) — not the earlier one

`.` (Dot) — Matches ANY Character Except Newline

text = 'This is Python\n class.'
pattern = r'.'
res = re.finditer(pattern, text)
# Matches every character EXCEPT the \n (newline is skipped)

pattern = r's.'                     # 's' followed by ANY character
text = 'This is\n Python\n class.'
res = re.finditer(pattern, text)
# Matches 's' + next char, e.g., 'si', 'ss' — but won't match 's' followed by \n

To match a LITERAL dot (not "any character"), escape it with \:

text = 'This. is P.ytho.n\n cla.ss.'
pattern = r'[.]'             # inside [] — . is treated literally
res = re.finditer(pattern, text)
# matches every actual '.' character

pattern = r'\.'                # backslash escape — also treats . literally
res = re.finditer(pattern, text)
# same result as [.] above

Escaping Other Special Characters

text = '$**^*&GGIUGB jd^ibsji&&)71625 nce^i7 bhi 1.03 83.383 hdj'

re.finditer(r'\^', text)     # matches literal '^' character
re.finditer(r'\$', text)      # matches literal '$' character
re.finditer(r'\.', text)       # matches literal '.' character

Rule: Any metacharacter (., ^, $, [, ], etc.) needs a \ before it to be treated as a literal character instead of a special instruction.

🔤 Special Sequences

# \w — matches alphanumeric characters (A-Z, a-z, 0-9, _)
# \W — matches everything EXCEPT alphanumeric (opposite of \w)

# \s — matches whitespace characters ( , \t, \v, \n, \r)
# \S — matches everything EXCEPT whitespace (opposite of \s)

# \d — matches digits (0-9)
# \D — matches everything EXCEPT digits (opposite of \d)

# \b — matches a WORD BOUNDARY (edge of a word)
# \B — matches everything EXCEPT a word boundary

text = 'ndhAe82JE 7yt2vj*&(*^*&$*tvhDNDdviuy7t7BDH er8732ubj _ hdude7_ hj83798Y^&^bjno'

re.finditer(r'\w', text)     # every letter, digit, underscore
re.finditer(r'\W', text)      # every symbol/space — opposite of \w

re.finditer(r'\s', text)       # spaces, tabs, newlines, etc.
re.finditer(r'\S', text)        # everything that ISN'T whitespace

re.finditer(r'\d', text)         # every digit 0-9
re.finditer(r'\D', text)          # everything that ISN'T a digit

Carriage Return `\r` and Vertical Tab `\v`

print('python\rHello')      # \r moves cursor to START of line — 'Hello' overwrites 'python'
                              # Output appears as: Hello (overwrites visually in terminal)
print('python\vHello')        # \v vertical tab — moves down a line, keeps horizontal position

`\b` — Word Boundary

text = 'catherine bought a cat and named him Tomcat. The cat loves catherine'

pattern = r'\bcat'        # matches 'cat' ONLY at the start of a word
res = re.finditer(pattern, text)
# Matches: 'cat' (standalone), 'cat' (in 'cat'), but NOT the 'cat' inside 'Tomcat' (mid-word)
# catherine ALSO matches because 'cat' is at the start of that word too!

pattern = r'\Bcat'         # matches 'cat' ONLY when NOT at a word boundary (mid-word)
res = re.finditer(pattern, text)
# Matches the 'cat' INSIDE 'Tomcat' only

pattern = r'\bcat\b'        # 'cat' as a COMPLETE, standalone word only
res = re.finditer(pattern, text)
# Matches ONLY the standalone word 'cat' — not 'catherine', not 'Tomcat'

This is the key pattern for "find the exact word, not a substring of a longer word."

🔗 `|` — OR Operator

text = 'xyz.com abc.net xyz@gmail.com xyz.in abc.com abc.gov'
pattern = 'com|gov|in'           # matches 'com' OR 'gov' OR 'in' — wherever found
res = re.finditer(pattern, text)
for i in res:
    print(i)

`()` — Groups (Combine Multiple Patterns)

# Match abc.com, abc.gov, abc.in, abc.net, xyz.com, xyz.gov, xyz.in, xyz.net
text = 'xyz.com abc.net xyz@gmail.com xyz.in abc.com abc.gov'
pattern = '(abc|xyz)(.com|.gov|.in|.net)'
res = re.finditer(pattern, text)
for i in res:
    print(i)

# Match hat, cat, rat, bat, mat — two equivalent approaches:
text = 'hat matrbsi jbatsiucatvs ratvdibat mat hat'

# Approach 1 — character set
pattern = r'[A-Za-z]at'
# matches ANY single letter + 'at'

# Approach 2 — explicit group of options
pattern = r'(h|r|m|b)at'
# matches EXACTLY h, r, m, or b + 'at' — more precise/restrictive

🔢 Quantifiers

# {3}     — exactly 3 repetitions
# {3,5}   — minimum 3, maximum 5 repetitions
# ?       — 0 or 1 repetition (optional)
# *       — 0 or MORE repetitions
# +       — 1 or MORE repetitions

text = 'hat hdb726 87tVmatrb1si jbae3tsiu4catvs&^%80ratvdiba3tbjsbihmat bhjbdb hat'

re.finditer(r'[0-9]', text)         # every SINGLE digit, one at a time

re.finditer(r'[0-9]{2}', text)       # groups of EXACTLY 2 consecutive digits
re.finditer(r'[0-9]{3}', text)        # groups of EXACTLY 3 consecutive digits

text2 = 'hat hdb726 7363 373773 8378733 8376t3 6333b77 93 873 8373777 3638273636763'
re.finditer(r'[0-9]{3,5}', text2)      # groups of 3 to 5 consecutive digits

# ? — 0 or 1
text3 = 'bavcsabcxsa'
re.finditer(r'a?', text3)              # matches 'a' if present, or empty string otherwise

# * — 0 or more
text4 = 'bavcsaaabcxsa'
re.finditer(r'a*', text4)                # matches runs of 'a's of ANY length (including zero)

# + — 1 or more
re.finditer(r'a+', text4)                 # matches runs of 'a's — but only where at least ONE exists

? vs * vs + — Quick Comparison:

Quantifier	Min	Max	Example match on `'aaa'`
`?`	0	1	`'a'` (just one, or none)
`*`	0	∞	`'aaa'` (the whole run, or empty)
`+`	1	∞	`'aaa'` (the whole run, requires ≥1)

☁️ DevOps / Cloud Use Cases (Regex)

import re

# 1. Validate an IP address pattern (basic)
text = 'Server connected from 192.168.1.105 on port 8080'
pattern = r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
ip = re.search(pattern, text)
print(ip.group())          # '192.168.1.105'

# 2. Extract all ERROR lines from a log
log = 'INFO: started\nERROR: disk full\nINFO: retry\nERROR: connection lost'
errors = re.findall(r'ERROR:.*', log)
print(errors)               # ['ERROR: disk full', 'ERROR: connection lost']

# 3. Extract port numbers from config text
config_text = 'app_port=8080, db_port=5432, cache_port=6379'
ports = re.findall(r'\d+', config_text)
print(ports)                 # ['8080', '5432', '6379']

# 4. Validate email format (basic pattern)
email = 'devops.engineer@company.com'
pattern = r'^[\w.]+@[\w]+\.[a-z]+$'
match = re.match(pattern, email)
print('Valid email' if match else 'Invalid email')

# 5. Extract resource names matching naming convention (env-service-number)
text = 'Resources: prod-web-01, staging-db-02, prod-cache-03'
pattern = r'\w+-\w+-\d{2}'
resources = re.findall(pattern, text)
print(resources)              # ['prod-web-01', 'staging-db-02', 'prod-cache-03']

# 6. Find lines starting with a timestamp pattern in logs
log_line = '2026-06-21 14:32:01 ERROR Connection timeout'
pattern = r'^\d{4}-\d{2}-\d{2}'
ts = re.match(pattern, log_line)
print(ts.group() if ts else 'No timestamp found')   # '2026-06-21'

❌ Common Mistakes (Regex)

Mistake	Issue	Fix
Using `match()` expecting full-string search	`match()` only checks from position 0	Use `search()` to find anywhere in the string
Forgetting case sensitivity	`'India'` ≠ `'india'` in default regex	Use `re.IGNORECASE` flag if needed
Treating `.` as a literal dot	`.` matches ANY character, not just `.`	Escape with `\.` or use `[.]`
Confusing `^` inside vs outside `[]`	Outside = "start of string"; inside = "NOT"	Context matters — memorize both meanings
Forgetting `r''` raw string prefix	Backslash sequences may be misinterpreted	Always use `r'pattern'` for regex
Using `findall()` when you need position info	`findall()` gives only strings, no span	Use `finditer()` for match objects with span/start/end
`\b` vs `\B` mix-up	`\b` = word boundary, `\B` = NOT a word boundary	`\bcat\b` for exact word; `\Bcat` for mid-word matches

🎯 Interview Points (Regex)

"Difference between re.match() and re.search()?"
→ match() only checks for a match at the very beginning of the string. search() scans the entire string and returns the first match found anywhere.
"Difference between findall() and finditer()?"
→ findall() returns a list of matched strings. finditer() returns an iterator of match objects, which gives access to span(), start(), end() for each match.
"What does . mean in regex, and how do you match a literal dot?"
→ . matches any character except a newline. To match a literal ., escape it as \. or place it inside a character set [.].
"What's the difference between \b and \B?"
→ \b matches a word boundary (start/end of a word). \B matches everywhere that is NOT a word boundary — useful for finding patterns strictly inside a word.
"Difference between *, +, and ? quantifiers?"
→ * = 0 or more, + = 1 or more (requires at least one), ? = 0 or 1 (optional, single occurrence).
"Why use raw strings (r'...') for regex patterns?"
→ To prevent Python from interpreting backslash sequences (like \n, \b) as escape characters before the regex engine sees them — ensures the pattern is passed through literally.

📚 Knowledge Base — Quick Revision (Regex)

import re

# ── FUNCTIONS ────────────────────────────────────────────────
re.match(pattern, text)        # only checks the START of string
re.search(pattern, text)        # first match ANYWHERE
re.findall(pattern, text)        # ALL matches → list of strings
re.finditer(pattern, text)        # ALL matches → iterator of match objects

# match object attributes
match.group()        # matched text
match.span()           # (start, end) tuple
match.start()            # start index
match.end()                # end index

# ── METACHARACTERS ──────────────────────────────────────────
.        # any character except \n
^        # start of string (outside []) / NOT (inside [])
$        # end of string
[]       # character set, e.g. [a-z], [0-9], [A-Za-z0-9]
[^...]   # NOT these characters
|        # OR
()       # grouping

# ── SPECIAL SEQUENCES ───────────────────────────────────────
\w  \W     # word char / NOT word char
\s  \S     # whitespace / NOT whitespace
\d  \D     # digit / NOT digit
\b  \B     # word boundary / NOT word boundary

# ── QUANTIFIERS ─────────────────────────────────────────────
{n}        # exactly n
{n,m}      # between n and m
?          # 0 or 1
*          # 0 or more
+          # 1 or more

# ── ALWAYS USE RAW STRINGS ──────────────────────────────────
pattern = r'\d+'     # ✅ raw string for regex patterns

🏋️ Practice Questions

Easy

Write a try-except-else-finally block that asks for two numbers, divides them, and demonstrates all 4 blocks running (success case) — print a distinct message in each.
Use re.findall() to extract all digits from the string: 'Server-01 has 8 CPUs and 32 GB RAM, port 8080'.
Write a regex pattern using \b to find the standalone word 'log' in the text: 'log file blogger logger log' (should match only the exact word 'log', not 'blogger' or 'logger').

Medium

Write a function safe_divide(a, b) that uses raise ValueError if b is zero, instead of letting Python's natural ZeroDivisionError happen. Catch it with a try-except and print a custom message.
Given a log string with multiple lines, use re.findall() with the pattern r'ERROR:.*' to extract all error messages. Then use finditer() on the same pattern to also print the span() of each match.
Write a regex pattern that extracts all valid AWS-style resource tags in this format: env-service-number (e.g., prod-web-01) from a string containing mixed text.

DevOps-Focused

Robust Deployment Function: Write a function deploy(version) that:
- Uses try to attempt the "deployment" (simulate with 1/int(version))
- Uses except to catch and print errors
- Uses else to print "Deployment successful" only if no error occurred
- Uses finally to always print "Deployment process finished"

Test it with deploy(2) and deploy(0).

Log Parser with Regex: Given this log data:

   log_data = '''2026-06-20 10:15:32 INFO Service started
   2026-06-20 10:16:45 ERROR Connection refused on port 5432
   2026-06-20 10:17:02 WARN High memory usage 85%
   2026-06-20 10:18:10 ERROR Disk space critical'''

Use regex to: (a) extract all timestamps using pattern r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}', (b) extract all ERROR messages using r'ERROR.*', (c) count total ERROR occurrences using len() on the findall() result.

Modules (Deep Dive), File Handling, Exception Handling

Tejas Shinkar — Sun, 21 Jun 2026 15:01:49 +0000

📌 Key Concepts Overview

Concept	One-Line Definition
Module	A `.py` file containing reusable functions/variables
`dir(module)`	Lists all functions/attributes available in a module
`sys.path`	List of directories Python searches when importing a module
File	Any container that stores code/data for reuse
Text vs Binary file	Text = human-readable (.txt, .py); Binary = everything else
File mode	Controls how a file is opened — `r`, `w`, `a`, `r+`, `w+`, `a+`
Cursor	Pointer tracking current read/write position in a file
`with open()`	Context manager — auto-closes the file, even on error
Exception	A runtime error that can crash a program if unhandled
`try-except`	Catches runtime errors so the program continues gracefully

📦 Part 1 — Modules (Deep Dive)

What's Inside a Module

import math
print(dir(math))     # lists every function/constant available in the math module

math.ceil(5/2)        # 3 — rounds UP
math.factorial(5)     # 120
math.pow(2, 3)        # 8.0 — note: returns FLOAT, unlike ** which can return int
math.sin(0.45)
angle = math.radians(0.45)

⚠️ Common Mistake: Calling ceil(5/2) WITHOUT importing math first → NameError: name 'ceil' is not defined. You must import math and use math.ceil(...).

3 Ways to Import

1. Import the whole module — access via module.function()

import math
math.pow(2, 3)
math.sqrt(5)

2. Import specific functions — access directly, no prefix needed

from math import pow
pow(2, 3)              # ✅ works directly

from math import pow, factorial, sqrt, ceil
sqrt(5)
factorial(6)
ceil(5/2)               # ✅ now works without math. prefix

3. Import everything with * — use ANY function directly

from math import *
sin(0.5)               # works without prefix

# ⚠️ DANGER: if two modules have a function with the SAME name,
# the second import SILENTLY OVERWRITES the first
from numpy import *
from pandas import *
# If both have a function called 'mean', whichever was imported LAST wins
# This can cause confusing bugs — avoid `import *` in real projects

`dir()` — See What's Currently Available in Your Namespace

print(dir())          # shows ALL names currently defined/imported in your session

import math
print(dir())           # now includes 'math'

from math import pow
print(dir())            # now ALSO includes 'pow' directly

Best Practice for DevOps scripts: Prefer import module_name and call module.function(). It's explicit, avoids name collisions, and makes code easier to read/maintain — critical when multiple engineers work on the same automation scripts.

Renaming a Module on Import (Aliasing)

import pandas as pd      # 'pd' is now the alias — standard convention
import boto3              # AWS SDK for Python — you'll use this constantly later
import requests            # for making HTTP API calls

These three are core DevOps/Cloud libraries:

Library	Used for
`boto3`	AWS SDK — create/manage EC2, S3, IAM, etc. from Python
`requests`	Calling REST APIs, webhooks, health-check endpoints
`pandas`	Processing CSV/log data, reporting

Creating a User-Defined Module

# Step 1: create a .py file (e.g., module.py) with function definitions:
# def square(x):
#     return x ** 2

# Step 2: import it like any other module
import module
module.square(2)         # 4

`sys.path` — Where Python Looks for Modules

import sys
print(sys.path)           # list of directories Python searches for modules

# If your module isn't in any of these folders, import fails with ModuleNotFoundError
# Fix: add the folder to the search path
sys.path.append(r"C:\Users\Lenovo\Desktop")
print(sys.path)            # now includes your custom folder

import module1              # now Python can find it

DevOps relevance: Understanding sys.path matters when organizing automation projects with custom helper modules (e.g., aws_utils.py, log_parser.py) that need to be importable from your main scripts.

📁 Part 2 — File Handling

What Is a File?

A file is any container used to store code or data for reuse — could be .txt, .py, .csv, .json, .log, images, audio, etc.

Two Categories in Python

Type	Examples	Description
Text files	`.txt`, `.py`, `.csv`, `.log`	Human-readable
Binary files	images, audio, video, `.pdf`, `.xlsx`	Everything else

The 4 File Operations

Create → Open
Read   → read()
Update → write() / append
Delete → close (releases the file resource)

Opening & Closing a File

# Syntax: file_handle = open('file_name', 'mode')
f = open('test.txt', 'r')
# ... do something ...
f.close()        # MUST close — otherwise risk losing unsaved data / locked file

Key File Functions

Function	Purpose
`write()`	Write data to the file
`read()`	Read data from the file
`tell()`	Returns current cursor position (index)
`seek(n)`	Moves cursor to position `n`

🔑 Part 3 — File Modes (THE Core of This Class)

`'r'` — Read Mode (Default)

f = open('test.txt', 'r')
print(f.tell())       # 0 — cursor starts at beginning
print(f.read())        # prints file content
print(f.tell())         # now at end of file (cursor moved after reading)
f.close()

Rules:

File must already exist → else FileNotFoundError
Can READ, cannot WRITE
Cursor starts at position 0 (beginning)

f = open('test.txt', 'r')
f.write('Trying to write...')    # ❌ io.UnsupportedOperation: not writable
f.close()

`'w'` — Write Mode

f = open('test.txt', 'w')
f.write('Everything has been removed from the file')
f.close()

Rules:

If file exists → WIPES all previous content (overwrites completely!)
If file doesn't exist → creates a new one
Can WRITE, cannot READ

f = open('test.txt', 'w')
f.write('New content')
print(f.read())        # ❌ io.UnsupportedOperation: not readable
f.close()

⚠️ Biggest danger of 'w' mode: Opening an existing file in 'w' mode instantly destroys its old content — even before you write anything new. Use with extreme caution on important files (this is exactly how production log files or configs get accidentally wiped).

`'a'` — Append Mode

f = open('test.txt', 'a')
print(f.tell())          # cursor starts at END of existing content (not 0!)
f.write('This is Python class')
print(f.tell())            # moved further right after writing
f.close()

Rules:

If file exists → cursor placed at the END; new writes are appended, old content preserved
If file doesn't exist → creates a new file
Can WRITE, cannot READ
seek() does NOT work for repositioning writes — append mode always writes at the end regardless of where you seek() to

f = open('test.txt', 'a')
f.seek(20)                    # tries to move cursor to position 20
f.write('This is weekend batch')   # still gets appended at the TRUE end, not position 20
f.close()

`'r+'` — Read AND Write Mode

f = open('test.txt', 'r+')
print(f.tell())          # 0
print(f.read())           # reads full content, cursor moves to end
f.close()

Rules:

File must exist → else FileNotFoundError
Can BOTH read and write
seek() works — lets you reposition before writing
⚠️ Writing OVERWRITES characters starting at the cursor position (doesn't insert — it replaces existing characters character-by-character)

f = open('test.txt', 'r+')
f.seek(3)
f.write('hey')     # overwrites whatever 3 characters started at position 3
f.close()

`'w+'` — Write AND Read Mode

f = open('test.txt', 'w+')
f.write('This is Python class')
print(f.tell())            # cursor is now at the end
f.seek(0)                   # must seek back to 0 to read what you just wrote!
print(f.read())              # 'This is Python class'
f.close()

Rules:

If file exists → wipes content (just like 'w')
If file doesn't exist → creates new
Can BOTH write and read — but you must seek(0) after writing if you want to read what you just wrote (cursor is at the end after writing)

`'a+'` — Append AND Read Mode

with open('test.txt', 'a+') as f:
    print(f.tell())                       # at end (existing content length)
    f.write('\nThis is weekend batch')     # appends
    print(f.tell())
    f.seek(0)                               # go back to start to read everything
    print(f.tell())
    print(f.read())                          # full file content, old + new

Rules:

Preserves existing content (like 'a')
Can read too — but must seek(0) first
seek() does NOT work to reposition WRITES — always appends at true end

📊 Mode Comparison Table (Memorize This)

Mode	File must exist?	Existing content	Read?	Write?	Cursor start
`r`	Yes (else error)	Preserved	✅	❌	Beginning
`w`	No (creates)	Wiped	❌	✅	Beginning
`a`	No (creates)	Preserved	❌	✅	End
`r+`	Yes (else error)	Preserved (overwritten on write)	✅	✅	Beginning
`w+`	No (creates)	Wiped	✅	✅	Beginning
`a+`	No (creates)	Preserved	✅	✅	End

🛡️ `with open()` — The Recommended Way

# Same as open(), but AUTO-CLOSES the file when the block ends — even if an error occurs
with open('test.txt', 'r') as f:
    print(f.tell())
    print(f.read())
    print(f.tell())
print(f.closed)        # True — automatically closed once outside the 'with' block
print(f.read())          # ❌ ValueError: I/O operation on closed file

Why this matters in DevOps: Forgetting f.close() in a long-running automation script can leak file handles, lock files on the OS, or lose buffered writes if the script crashes. with open() eliminates this entire class of bugs — always prefer it over manual open()/close().

🔧 Part 4 — Functions on a File

`read()` — Read Entire Content as One String

with open('test.txt', 'r') as f:
    print(f.tell())          # 0
    data = f.read()
    print(f.tell())           # moved to end

print(data, type(data))        # full content, <class 'str'>

`readline()` — Read ONE Line at a Time

with open('test.txt', 'r') as f:
    print(f.tell())
    print(f.readline(), type(f.readline()))   # reads from cursor up to end of THAT line
    print(f.tell())

# seek() lets you jump to a specific position before reading a line
with open('test.txt', 'r') as f:
    f.seek(26)
    print(f.readline())        # reads the rest of the line starting at position 26

`readlines()` — Read ALL Lines as a LIST

with open('test.txt', 'r') as f:
    data = f.readlines()

print(data, type(data))     # ['line1\n', 'line2\n', ...] <class 'list'>

for i in data:
    print(i)                  # iterate line by line

read() vs readline() vs readlines():

Method	Returns	Use case
`read()`	One big string (whole file)	Small files, need full text
`readline()`	One string (single line)	Process huge files line-by-line (memory efficient)
`readlines()`	A list of strings (each line)	Need to loop/count/index by line

`write()` — Write a String

with open('test.txt', 'w+') as f:
    f.write('This is Python Class')
    f.write('\nThis is Devops batch')     # \n needed for new line — write() does NOT auto-newline
    f.write('\nThis is weekend Class')
    f.seek(0)
    print(f.read())

`writelines()` — Write a LIST of Strings

lst = ['Python\n', 'DevOps\n', 'Linux\n', 'AWS\n']
with open('test.txt', 'w+') as f:
    f.writelines(lst)            # writes each list item — \n must be included manually
    f.seek(0)
    print(f.read())

⚠️ writelines() does NOT add newlines automatically — if your list items don't have \n, everything gets written on one continuous line.

☁️ DevOps / Cloud Use Cases

# 1. Reading a server config file
with open('server_config.txt', 'r') as f:
    config = f.read()
    print(config)

# 2. Appending to a log file (most common DevOps file pattern)
with open('deployment.log', 'a+') as f:
    f.write('\n[INFO] Deployment started at 14:30')

# 3. Writing a fresh report each run
with open('health_report.txt', 'w') as f:
    f.write('Server Health Report\n')
    f.write('All systems operational\n')

# 4. Reading line-by-line for large log files (memory efficient)
with open('access.log', 'r') as f:
    for line in f:
        if 'ERROR' in line:
            print(line.strip())

# 5. Counting errors in a log file
with open('app.log', 'r') as f:
    lines = f.readlines()
error_count = sum(1 for line in lines if line.startswith('ERROR'))
print(f'Total errors: {error_count}')

# 6. Writing a list of server names to a file (inventory)
servers = ['web-01\n', 'web-02\n', 'db-01\n']
with open('inventory.txt', 'w') as f:
    f.writelines(servers)

⚠️ Part 5 — Errors: Two Types

1. Syntax Error (Compile-time error)
   — caught BEFORE the program runs
   — e.g., missing colon, unmatched quotes

2. Runtime Error (Logical / Exception)
   — happens WHILE the program runs
   — e.g., dividing by zero, file not found, wrong data type

print('Devops)        # ❌ SyntaxError — unclosed string, caught immediately, nothing runs

print('Hello')
print(10/0)             # ❌ ZeroDivisionError — this is a RUNTIME error
print('Code working fine')   # never reached — program crashes at the error above

Key takeaway: A syntax error stops the ENTIRE program from running, even lines before it. A runtime error stops execution AT that line — lines before it already ran successfully.

🧯 Part 6 — Exception Handling

Why Exception Handling?

To handle runtime errors gracefully so the program doesn't crash suddenly — especially important for automation scripts that run unattended (cron jobs, CI/CD pipelines, monitoring scripts).

Basic `try-except`

a = int(input('Enter a number: '))
b = int(input('Enter the second number: '))
try:
    print(a/b)                          # suspicious code goes here
except:
    print("Denominator can't be zero")   # runs ONLY if try block raises an error
print('Hello')                            # this STILL runs — program didn't crash!

Multiple Statements Inside `try`

try:
    a = int(input('Enter a number: '))
except:
    print('Only integers are allowed')

print('Hello')

try:
    print(a/b)
except:
    print("Denominator can't be zero")
print('Hello')

Catching the Actual Exception with `as e`

try:
    a = int(input('Enter a number: '))
    b = 10/a
    lst = [1,2,3,4,5]
    print(lst[a])
except Exception as e:
    print(e)              # prints the ACTUAL error message — much more useful for debugging

print(dir(Exception))     # see all exception attributes/methods

Specific Exception Types — Best Practice

try:
    a = int(input('Enter a number: '))
except ValueError:
    print('Only numbers are allowed')

try:
    b = 10/a
except ZeroDivisionError:
    print('Cannot divide by zero')

try:
    lst = [1,2,3,4,5]
    print(lst[a])
except IndexError:
    print('The index is out of range')

Why use specific exceptions instead of bare except:?

Bare except: catches EVERYTHING — including bugs you didn't anticipate, masking real problems
Specific exceptions (ValueError, ZeroDivisionError, IndexError, FileNotFoundError) let you handle each error type appropriately and give precise feedback

Common Exception Types Seen in Class

Exception	Triggered by
`ValueError`	Casting invalid data, e.g. `int('abc')`
`ZeroDivisionError`	Dividing by zero
`IndexError`	Accessing a list index that doesn't exist
`FileNotFoundError`	Opening a non-existent file in `'r'` mode
`Exception`	Generic catch-all (base class for most exceptions)

# FileNotFoundError example
with open('test11.txt', 'r') as f:    # file doesn't exist
    print(f.read())
# ❌ FileNotFoundError: [Errno 2] No such file or directory: 'test11.txt'

☁️ DevOps / Cloud Use Cases (Exception Handling)

# 1. Resilient config file reader — won't crash the whole automation script
try:
    with open('config.txt', 'r') as f:
        config = f.read()
except FileNotFoundError:
    print('⚠️ Config file missing — using defaults')
    config = 'default_config'

# 2. Safe API call handling (will be used with requests later)
import requests
try:
    response = requests.get('https://api.example.com/health', timeout=5)
    print(response.status_code)
except requests.exceptions.RequestException as e:
    print(f'API call failed: {e}')

# 3. Safe division for resource utilisation percentage
try:
    used = 80
    total = 0
    percentage = (used / total) * 100
except ZeroDivisionError:
    print('⚠️ Total capacity is zero — check configuration')

# 4. Validating user-provided port number
try:
    port = int(input('Enter port: '))
except ValueError:
    print('❌ Port must be a number')
    port = None

# 5. Retry loop combined with exception handling (polling pattern)
import time
for attempt in range(3):
    try:
        with open('status.txt', 'r') as f:
            status = f.read()
        break
    except FileNotFoundError:
        print(f'Attempt {attempt+1}: status file not ready yet')
        time.sleep(2)

❌ Common Mistakes

Mistake	Code	Fix
Reading in `'w'` mode	`open('f.txt','w'); f.read()`	`'w'` mode is write-only — use `'r+'` or `'w+'`
Writing in `'r'` mode	`open('f.txt','r'); f.write(...)`	`'r'` mode is read-only — use `'w'`, `'a'`, or `'r+'`
Forgetting `seek(0)` after writing in `w+`/`a+`	Write then immediately `read()` → gets empty string	`f.seek(0)` before reading
Using `'w'` on an important file	Instantly wipes existing content	Use `'a'` to preserve data, or back up first
`seek()` in append mode	Expecting `seek()` to control WHERE append writes	Append mode always writes at the true end regardless
Bare `except:` swallowing all errors	Hides real bugs, hard to debug	Catch specific exceptions: `except ValueError:`
Forgetting `f.close()` (without `with`)	File handle stays open, risk of data loss	Always use `with open(...) as f:`
`writelines()` without `\n`	All text runs together on one line	Include `\n` in each list item
SyntaxError stops everything	A typo like `'Devops)` halts the whole program	Fix the syntax before runtime errors even matter

🎯 Interview Points

"Difference between import module and from module import function?"
→ import module requires module.function() calls — explicit and avoids name collisions. from module import function lets you call function() directly but risks overwriting names from other imports.
"Why avoid from module import *?"
→ It pollutes the namespace and can silently overwrite functions/variables from other modules with the same name, causing hard-to-debug bugs.
"Difference between 'r+' and 'w+' mode?"
→ 'r+' requires the file to exist and preserves content. 'w+' creates the file if missing and wipes existing content immediately upon opening.
"Why use with open() instead of open() + close()?"
→ with guarantees the file is closed automatically — even if an exception occurs inside the block — preventing resource leaks and data loss.
"Difference between read(), readline(), and readlines()?"
→ read() returns the whole file as one string. readline() returns one line at a time (memory efficient for huge files). readlines() returns a list of all lines.
"Why does seek() not work for repositioning writes in append mode?"
→ Append mode ('a'/'a+') is OS-level guaranteed to always write at the true end of the file, regardless of cursor position — this prevents accidental data corruption from concurrent writers.
"What's the difference between a syntax error and a runtime error?"
→ Syntax errors are caught before execution begins (e.g., missing colon) and stop the program entirely. Runtime errors occur during execution (e.g., division by zero) and only affect code from that point onward — code before it has already run.
"Why use specific exceptions (ValueError, ZeroDivisionError) instead of a bare except:?"
→ Specific exceptions let you handle different failure modes appropriately and avoid silently catching bugs you didn't anticipate. Bare except: can mask real problems.
"What's the difference between except: and except Exception as e:?"
→ except: catches the error but gives you no information about it. except Exception as e: captures the actual exception object so you can inspect/print/log the real error message.

📚 Knowledge Base — Quick Revision

# ── MODULE IMPORTS ──────────────────────────────────────────
import math                      # math.func()
from math import sqrt            # sqrt() directly
from math import *                # everything directly (avoid in real code)
import pandas as pd                # aliasing
import sys; sys.path.append(dir)   # add custom module search path

# ── FILE MODES ──────────────────────────────────────────────
'r'    # read only, file must exist
'w'    # write only, WIPES existing content, creates if missing
'a'    # write only, APPENDS at end, creates if missing
'r+'   # read + write, file must exist, cursor at start
'w+'   # read + write, WIPES content, creates if missing
'a+'   # read + write, APPENDS, creates if missing

# ── FILE OPERATIONS ─────────────────────────────────────────
with open('file.txt', 'mode') as f:   # auto-closes — ALWAYS prefer this
    f.read()          # whole file as one string
    f.readline()       # one line at a time
    f.readlines()       # list of all lines
    f.write('text')      # write a string (no auto \n)
    f.writelines(lst)     # write a list of strings (no auto \n)
    f.tell()                # current cursor position
    f.seek(n)                 # move cursor to position n

# ── EXCEPTION HANDLING ──────────────────────────────────────
try:
    risky_code()
except SpecificError:
    handle_it()
except Exception as e:
    print(e)            # see the actual error

# Common exceptions:
# ValueError        — bad type conversion
# ZeroDivisionError — division by zero
# IndexError        — list index out of range
# FileNotFoundError — file doesn't exist (in 'r' mode)

🏋️ Practice Questions

Easy

Open a file notes.txt in write mode, write 3 lines (using \n), close it, then reopen in read mode and print the content.
What's the difference between f.read() and f.readlines()? Demonstrate both on the same file.
Write a try-except block that safely converts user input to an integer, catching ValueError if the input isn't numeric.

Medium

Create a file log.txt using 'a+' mode. Append 3 log lines on 3 separate runs (simulate by calling the write block 3 times). Read and print the full file after the 3rd append.
Write a script using try-except with THREE specific except blocks (ValueError, ZeroDivisionError, IndexError) that takes a number from the user, divides 100 by it, and accesses that index in a 5-element list.
Explain (with code) why opening an existing file in 'w' mode and immediately closing it without writing anything still destroys the original content.

DevOps-Focused

Log File Analyzer: Create a file server.log with writelines() containing a mix of INFO, WARN, and ERROR lines. Then read it back with readlines() and count how many lines start with each level. Wrap the file-reading part in a try-except FileNotFoundError block.
Safe Config Loader: Write a function load_config(filename) that tries to open and read a config file. If the file doesn't exist, catch FileNotFoundError and return a default config string instead of crashing. Test it with both an existing and a non-existing filename.

Functions, Scope (Local/Global), Arguments, return, Lambda, filter()/map()/reduce(), Modules

Tejas Shinkar — Sun, 21 Jun 2026 14:20:36 +0000

📌 Key Concepts Overview

Concept	One-Line Definition
Function	A reusable, named block of code that runs when called
Parameter vs Argument	Parameter = name in definition. Argument = value passed at call time
Local variable	Exists only inside the function it's defined in
Global variable	Exists outside any function, accessible everywhere
`global` keyword	Lets a function modify a global variable
`return`	Sends a value back from the function and ends it
`*args`	Collects extra positional arguments into a tuple
`**kwargs`	Collects extra keyword arguments into a dict
`lambda`	A small, unnamed (anonymous) one-line function
`filter()`	Keeps only elements that satisfy a condition
`map()`	Applies an operation to every element
`reduce()`	Aggregates an iterable into a single value
Module	A `.py` file containing reusable functions/classes

🧩 Part 1 — Why Functions?

# WITHOUT a function — code duplicated everywhere
a, b = 10, 20
print(a + b)
str1 = input('Enter a string: ')
if str1 == str1[::-1]:
    print('Palindrome')

# ... later in the program, same logic copy-pasted again ...

Functions exist to:

Avoid repeating code (write once, call many times)
Reduce program size
Organize logic into reusable, testable units

def add(a, b):
    c = a + b
    print(c)

def is_palindrome():
    str1 = input('Enter a string: ')
    if str1 == str1[::-1]:
        print('Palindrome')
    else:
        print('Not a Palindrome')

# Now reuse anywhere:
add(327, 46)
is_palindrome()

Two Types of Functions

Type	Example
In-built (Python provides)	`print()`, `len()`, `type()`, `input()`
User-defined (you write)	`def add(a, b): ...`

The 2 Components of Every Function

Function Definition — written using def, contains the logic. Does NOT run on its own.
Function Call — function_name() — this is what actually executes the code.

def add():            # ← definition (just creates it, doesn't run it)
    print('Hello')

add()                  # ← call (THIS runs it)

⚠️ Don't Shadow Built-in Functions (Recap + New Example)

# ❌ DANGEROUS — overwrites Python's built-in print/len
def print():
    print('Hello')      # this would now infinitely recurse / break

def len():
    print('Hello')

len('Python')            # ❌ TypeError — len() no longer works as expected,
                          # it now expects ZERO arguments because you redefined it

Lesson: Never name your functions print, len, list, input, type, etc. — you permanently break the built-in within that scope.

🎯 Parameterised vs Non-Parameterised Functions

# Non-parameterised — takes no input, asks inside the function
def add():
    a = int(input('Enter a number: '))
    b = int(input('Enter the next number: '))
    c = a + b
    print(c)

# Parameterised — takes input directly via parameters
def add1(x, y):
    c = x + y
    print(c)

add1(18, 46)     # 18 and 46 are ARGUMENTS passed to PARAMETERS x and y

Parameter vs Argument — exact definition:

Term	Where it appears	Example
Parameter	In the function definition	`def add1(x, y):` → `x`, `y`
Argument	In the function call	`add1(18, 46)` → `18`, `46`

def even_odd(x):
    if x % 2 == 0:
        print('Even')
    else:
        print('odd')

even_odd(37)        # 'odd'

☁️ DevOps-Flavoured Function Examples

def restart_service(service_name):
    print(f"Restarting {service_name} service......")
    print(f"{service_name} restarted successfully")

restart_service("nginx")
restart_service("Docker")

def check_disk_usage():
    print("Checking disk usage...")
    print("Disk usage is 75%")

# Real system command execution
import os
def check_disk():
    os.system("df -h")        # runs actual shell command

lists have .append() but strings do NOT — str1.append('xyz') → AttributeError. Strings are immutable, so there's no in-place method to add to them; use concatenation or += instead.

🔒 Part 2 — Local vs Global Variables

Local Variables

def add(a, b):           # a, b, c are LOCAL — only exist inside this function
    c = a + b
    print(c)

print(a)        # ❌ NameError: name 'a' is not defined — 'a' doesn't exist outside the function
print(c)        # ❌ NameError — same reason

Global Variables

x = 47          # global variable
y = 48          # global variable

def add1(x, y):       # these x, y are LOCAL parameters — shadow the global ones
    c = x + y
    print(c)

add1(x, y)             # add1(47, 48) → prints 95
print(x)               # 47 — global x is untouched, local x inside function was separate

Key rule: A local variable and a global variable can share the same name — they are still two completely different variables. The function's local one "shadows" the global one only inside the function body.

Reading a Global Variable Inside a Function — Works Fine

count = 10
def update1():
    global count
    print("Inside function:", count)   # can READ global without 'global' keyword too

update1()         # Inside function: 10

Modifying a Global Variable Without `global` — Creates a NEW Local Variable

count = 10
def update():
    count = 20                          # this creates a NEW local 'count'
    print("Inside function:", count)    # Inside function: 20

update()
print(count)        # 10 — the GLOBAL count is untouched!

The `global` Keyword — Actually Modify the Global Variable

count = 10
def update2():
    global count          # tells Python: use the OUTER 'count', don't create a local one
    count = 20
    print("Inside function:", count)

update2()
print(count)        # 20 — global count WAS modified this time

The #1 confusion point in this topic:

Code	Effect on global variable
`count = 20` (no `global`) inside function	Creates a separate LOCAL variable — global untouched
`global count` then `count = 20`	Modifies the actual GLOBAL variable

↩️ Part 3 — `return` Statement

Print vs Return — The Critical Difference

def add1(x, y):
    c = x + y
    return c              # sends value BACK to the caller

result1 = add1(10, 20)    # result1 now HOLDS the value 30
print(result1)             # 30
print(result1 ** 2)        # 900 ✅ — can use the result further

def add2(x, y):
    c = x + y
    print(c)               # just prints — doesn't send value back

result2 = add2(10, 20)     # prints 30, but returns nothing
print(result2)              # None  ⚠️ — function with no return gives None
print(result2 ** 2)         # ❌ TypeError: unsupported operand for None

Golden Rule: Use print() inside a function only for display. Use return when you need to use the result later in your program (further calculations, conditions, storing in variables/lists).

`return` Terminates the Function Immediately

def add1(x, y):
    c = x + y
    return c
    print('Hello')        # ⚠️ NEVER runs — return already exited the function

def add2(x, y):
    c = x + y
    print('Hello')        # runs FIRST — this line is before return
    return c

add2(10, 20)               # prints 'Hello', then returns c (silently, unless captured)

Returning Multiple Values (as a Tuple)

def calc(x, y):
    c = x + y
    d = x - y
    return c, d            # returns BOTH — Python packs them into a tuple

t = calc(20, 10)
print(t, type(t))          # (30, 10) <class 'tuple'>

Practical Functions — Factorial & Prime Check

# Factorial
def factorial(n):
    f = 1
    for i in range(1, n + 1):
        f *= i
    return f

x = factorial(5)
print(x)            # 120

# Prime check
def is_prime():
    n = int(input('Enter a number: '))
    if n < 2:
        return False
    for i in range(2, n):
        if n % i == 0:
            return False
    else:
        return True

print(is_prime())

📥 Part 4 — Types of Arguments

1. Positional argument
2. Keyword argument
3. Default argument
4. Variable-length argument
     a. *args    → variable positional args (tuple)
     b. **kwargs → variable keyword args (dict)

1. Positional Arguments — Order Matters

def emp_details(name, age, salary):
    print('Name :- ', name)
    print('Age :- ', age)
    print('Salary :- ', salary)

emp_details('Sourav', 25, 75000)        # ✅ correct order
emp_details('Sourav', 75000, 25)        # ⚠️ runs but WRONG — age=75000, salary=25 (logic bug!)
emp_details(25, 75000, 'Sourav')        # ⚠️ runs but completely scrambled

emp_details('Sourav', 75000)            # ❌ TypeError: missing required argument 'salary'

Rule: Number of arguments must exactly match number of parameters, and order determines which value goes to which parameter.

2. Keyword Arguments — Name the Parameter Explicitly

emp_details('Sourav', 25, salary=75000)               # ✅ mix positional + keyword
emp_details('Sourav', age=25, salary=75000)            # ✅ all named after first
emp_details(salary=75000, age=25, name='Sourav')        # ✅ order doesn't matter when all keyworded

emp_details(salary=75000, age=25, Name='Sourav')        # ❌ TypeError — 'Name' ≠ 'name' (case-sensitive!)
emp_details(salary=75000, age=25, 'Sourav')              # ❌ SyntaxError — positional AFTER keyword not allowed

⚠️ Critical Rule: If mixing positional and keyword arguments, all positional arguments must come BEFORE any keyword arguments.

3. Default Arguments — Fallback Value if Not Provided

def emp_details(name, age, salary, loc='Bangalore'):    # loc has a default
    print('Name :- ', name)
    print('Age :- ', age)
    print('Salary :- ', salary)
    print('Location :- ', loc)

emp_details('Sourav', salary=75000, age=25)                          # loc defaults to 'Bangalore'
emp_details('Sourav', salary=75000, age=25, loc='Delhi')              # loc overridden to 'Delhi'

DevOps relevance: Default arguments are everywhere — region='ap-south-1', timeout=30, retries=3 — sensible defaults that can be overridden when needed.

4. Variable-Length Arguments

def add(x, y):
    return x + y

add(10, 38)              # ✅ works
add(37, 25, 58)           # ❌ TypeError — too many arguments, only 2 parameters defined

*args — Variable Positional Arguments → packed into a TUPLE

def xyz(*args):
    print(args, type(args))

xyz(10, 49, 28, 'Python')      # (10, 49, 28, 'Python') <class 'tuple'>

def add(*args):
    c = sum(args)
    return c

add(20, 49)                     # 69
add(18, 437, 28)                 # 483
add(37, 483, 746, 276, 373, 483, 27893)   # works with ANY number of arguments

`kwargs` — Variable Keyword Arguments → packed into a DICT**

def abc(**kwargs):
    print(kwargs, type(kwargs))

abc(a=10, b=20, c=30, d=40)
# {'a': 10, 'b': 20, 'c': 30, 'd': 40} <class 'dict'>

DevOps relevance: *args/**kwargs are everywhere in real tools — e.g., Boto3 functions, Django views, decorators — to accept flexible config without hardcoding every possible parameter.

⚡ Part 5 — Lambda (Anonymous Functions)

# Syntax: lambda arguments: expression
# Called immediately: (lambda arguments: expression)(call_arguments)

# Add two numbers
(lambda x, y: x + y)(20, 48)        # 68

a = (lambda x, y: x * y)(20, 20)
print(a)                              # 400

# Conditional (ternary) inside lambda
(lambda a, b, c: a if a > b and a > c else b if b > a and b > c else c)(28, 37, 18)
# 37 — finds the maximum of 3 numbers

(lambda x: 'Even' if x % 2 == 0 else 'Odd')(23)
# 'Odd'

When to use lambda: Quick, throwaway, single-expression functions — especially as arguments to filter(), map(), reduce(), or sorted(key=...). NOT meant for complex multi-line logic (use a regular def for that).

🔍 Part 6 — `filter()`, `map()`, `reduce()`

`filter()` — Keep Elements That Match a Condition

# filter(lambda argument: condition, iterable)
# Returns a filter OBJECT — wrap with list() or tuple() to see values

# Manual for-loop equivalent (what filter replaces):
lst = [1,2,3,4,5,6,7,8,9,10]
lst1 = []
for i in lst:
    if i % 2 == 0:
        lst1.append(i)
print(lst1)                          # [2,4,6,8,10]

# Same thing with filter() — one line
print(list(filter(lambda x: x % 2 == 0, lst)))     # [2,4,6,8,10]

# DevOps example: flag high CPU usage servers
cpu_usage = [48, 27, 93, 14, 86]
high_cpu = list(filter(lambda x: x > 80, cpu_usage))
print(high_cpu)                       # [93, 86]

# Extract vowels from a string
str1 = 'ncihvewc cnoejife2oiyfecndlncjihuhew'
print(tuple(filter(lambda x: x.lower() in 'aeiou', str1)))

`map()` — Apply an Operation to Every Element

# map(lambda argument: expression, iterable)
# Returns a map OBJECT — wrap with list() or tuple()

# Manual for-loop equivalent:
lst = [1,2,3,4,5,6,7,8,9,10]
lst1 = []
for i in lst:
    lst1.append(i**2)
print(lst1)                           # [1,4,9,16,25,36,49,64,81,100]

# Same with map() — one line
print(list(map(lambda x: x**2, lst)))   # [1,4,9,16,25,36,49,64,81,100]

# map() on heterogeneous list — operation applies to EVERY element
lst = [1, 2, 3, 'Python', 4, 5, 6, 7, 'DevOps', 8, 9, 10]
print(list(map(lambda x: x*2, lst)))
# [2,4,6,'PythonPython',8,10,12,14,'DevOpsDevOps',16,18,20]
# Note: x*2 means NUMERIC doubling for numbers, STRING repetition for strings

# map() on a string — applies to each character
str1 = "python"
print(list(map(lambda x: x*2, str1)))
# ['pp','yy','tt','hh','oo','nn']

filter() vs map() — Core Difference:

Function	Purpose	Output size
`filter()`	Keep only items matching a condition	≤ original (may shrink)
`map()`	Transform every item	= original (same size)

`reduce()` — Aggregate to a Single Value

from functools import reduce      # must import — not built-in like filter/map

# reduce(lambda x, y: expression, iterable)

lst = [1,2,3,4,5,6,7,8,9,10]
res = reduce(lambda x, y: x + y, lst)
print(res)                          # 55 — sum of all elements

# Find minimum
reduce(lambda x, y: x if x < y else y, lst)    # 1

# Find maximum
reduce(lambda x, y: x if x > y else y, lst)    # 10

How reduce() works internally: It takes the first two elements, applies the lambda, takes that result + the next element, applies the lambda again — repeating until one value remains.

📦 Part 7 — Modules

# A module = a .py file containing functions/classes
# Access with: module_name.function_name()

import Mod                  # imports Mod.py
Mod.greet()                  # calls greet() function inside Mod.py
Mod.adding(37, 27)
Mod.mul(37, 17)

Best practice: Don't put "free executable code" (code that runs immediately on import) inside a module — only function/class definitions. This keeps the module reusable and side-effect-free when imported elsewhere.

DevOps relevance: This is exactly how you'd structure a utils.py or aws_helpers.py module with reusable functions like get_ec2_status(), restart_service(), parse_log() — imported across multiple automation scripts.

☁️ DevOps / Cloud Use Cases

# 1. Reusable service restart function with default region
def restart_service(service_name, region='ap-south-1'):
    print(f"Restarting {service_name} in {region}...")

restart_service('nginx')
restart_service('nginx', region='us-east-1')

# 2. *args to accept any number of servers to health-check
def health_check(*servers):
    for s in servers:
        print(f'Checking {s}...')

health_check('web-01', 'web-02', 'db-01')

# 3. **kwargs to accept flexible AWS-style config
def launch_instance(**config):
    print(config)

launch_instance(instance_type='t2.micro', region='ap-south-1', count=2)

# 4. filter() to find unhealthy servers
server_status = {'web-01': 200, 'web-02': 500, 'db-01': 200, 'cache-01': 503}
unhealthy = list(filter(lambda s: server_status[s] != 200, server_status))
print(unhealthy)                      # ['web-02', 'cache-01']

# 5. map() to convert a list of instance IDs into ARNs
instance_ids = ['i-001', 'i-002', 'i-003']
arns = list(map(lambda i: f'arn:aws:ec2:ap-south-1:123456789:instance/{i}', instance_ids))
print(arns)

# 6. reduce() to sum total monthly AWS costs across services
from functools import reduce
service_costs = [120.50, 45.25, 300.10, 15.00]
total = reduce(lambda x, y: x + y, service_costs)
print(f'Total monthly cost: ${total}')

# 7. global + counter pattern for tracking retries across calls
retry_count = 0
def attempt_connection():
    global retry_count
    retry_count += 1
    print(f'Attempt #{retry_count}')

attempt_connection()
attempt_connection()
print(f'Total retries: {retry_count}')

❌ Common Mistakes

Mistake	Code	Fix
Confusing `print()` and `return`	Function uses `print()`, then tries `result = func()` and uses `result`	Use `return` when you need the value later
Forgetting `global` keyword	`count = 20` inside function — global stays unchanged	Use `global count` before modifying
Code after `return`	Lines after `return` never execute	Put cleanup/logging BEFORE `return`
Positional after keyword	`f(a=1, 2)`	All positional args must come first
Case-sensitive keyword names	`f(Name='x')` when param is `name`	Match exact parameter name and case
Shadowing built-ins with function names	`def len(): ...`	Never name functions after built-ins
Forgetting `list()`/`tuple()` around `filter`/`map`	`print(filter(...))` → shows object, not values	Wrap with `list()` or `tuple()`
Forgetting to `import functools` for `reduce`	`reduce(...)` → NameError	`from functools import reduce`
String `.append()`	`str1.append('x')`	Strings are immutable — use `+=` or concatenation

🎯 Interview Points

"Difference between parameter and argument?"
→ Parameter is the variable name in the function definition. Argument is the actual value passed when calling the function.
"Difference between local and global variables?"
→ Local variables exist only inside the function where they're defined. Global variables are defined outside any function and accessible everywhere — but a function can't modify a global variable without the global keyword.
"What does the global keyword do?"
→ It tells Python that an assignment inside a function should modify the OUTER (global) variable instead of creating a new local one.
"Difference between print() and return inside a function?"
→ print() just displays output to console — the function still returns None. return sends the value back to the caller so it can be stored or reused.
"What happens to code written after a return statement?"
→ It never executes — return immediately exits the function.
"Difference between *args and `kwargs?"** →args collects extra positional arguments into a tuple. *kwargs` collects extra keyword arguments into a dictionary.
"Difference between filter() and map()?"
→ filter() returns only the elements that satisfy a condition (output may be smaller). map() applies a transformation to every element (output is always the same size).
"Why do you need to import reduce but not filter/map?"
→ filter() and map() are built-in functions. reduce() lives in the functools module and must be explicitly imported.
"What is a lambda function and when would you use it?"
→ An anonymous, single-expression function — used for short, throwaway logic, typically as an argument to filter(), map(), reduce(), or sorted().
"Can a function return multiple values?"
→ Yes — Python packs them into a tuple automatically: return c, d → caller receives (c, d).

📚 Knowledge Base — Quick Revision

# ── FUNCTION BASICS ─────────────────────────────────────────
def func_name(param1, param2):     # definition
    ...
    return value                    # optional

func_name(arg1, arg2)               # call — this is what executes it

# ── SCOPE ───────────────────────────────────────────────────
x = 10                  # global
def f():
    x = 20               # LOCAL — does NOT affect global x
def g():
    global x
    x = 20               # modifies the actual global x

# ── ARGUMENT TYPES ──────────────────────────────────────────
def f(a, b):              pass    # positional
f(a=1, b=2)                       # keyword
def f(a, b=10):            pass   # default (b optional)
def f(*args):               pass  # variable positional → tuple
def f(**kwargs):             pass # variable keyword → dict

# Rule: positional args must come BEFORE keyword args in a call

# ── RETURN VS PRINT ─────────────────────────────────────────
def f():
    return 5      # value usable later: x = f()
def g():
    print(5)       # just displays; g() itself returns None

# ── LAMBDA ──────────────────────────────────────────────────
lambda x, y: x + y                              # basic
lambda x: 'Even' if x % 2 == 0 else 'Odd'        # ternary inside lambda

# ── FILTER / MAP / REDUCE ───────────────────────────────────
list(filter(lambda x: condition, iterable))      # keep matching items
list(map(lambda x: expression, iterable))        # transform every item
from functools import reduce
reduce(lambda x, y: expression, iterable)        # aggregate to ONE value

# ── MODULES ─────────────────────────────────────────────────
import module_name
module_name.function_name()

🏋️ Practice Questions

Easy

Write a function square(n) that takes a number and returns its square. Call it and print the result.
Write a function greet(name, greeting='Hello') with a default argument. Call it once with just a name, and once overriding the greeting.
Use lambda to write a one-line function that checks if a number is positive, negative, or zero.

Medium

Write a function total(*args) that accepts any number of numbers and returns their sum using reduce().
Demonstrate the difference between local and global scope: create a global variable counter = 0, write a function that tries to increment it without global, show it fails to update, then fix it using the global keyword.
Given cpu_list = [45, 78, 92, 33, 88, 67], use filter() to get servers with CPU > 70, and map() to convert all values to strings with a % symbol (e.g., '45%').

DevOps-Focused

Service Manager Function: Write a function manage_service(action, service_name, region='ap-south-1') that prints a message like "Performing 'restart' on nginx in ap-south-1". Call it 3 times: once with all positional args, once with keyword args in different order, and once overriding the region.
Cost Calculator with reduce(): Given a list of dicts representing AWS services and their monthly costs:

   services = [
       {'name': 'EC2', 'cost': 120.50},
       {'name': 'S3', 'cost': 45.25},
       {'name': 'RDS', 'cost': 300.10},
   ]

Use map() to extract just the costs into a list, then use reduce() (with functools) to calculate the total. Print: "Total AWS monthly cost: $465.85".

while Loop, break & continue, Lists (Creation, Mutability, Methods, List Comprehension)

Tejas Shinkar — Sat, 20 Jun 2026 15:19:43 +0000

📌 Key Concepts Overview

Concept	One-Line Definition
`while` loop	Repeats code as long as a condition is `True`
`while True`	Infinite loop — needs `break` to stop
`break`	Immediately exits the loop
`continue`	Skips current iteration, moves to next
List	Ordered, mutable collection — heterogeneous elements allowed
List Comprehension	One-line way to build a list using a loop + condition
List Mutability	Lists can be changed in place — `id()` stays the same

🔁 Part 1 — `while` Loop

The 3 Components (Critical Pattern)

# 1. Initialisation  2. Condition  3. Increment/Decrement

a = 1                  # 1. Initialisation
while a <= 10:          # 2. Condition
    print('Devops')
    a += 1              # 3. Increment

# Without increment → INFINITE LOOP (condition never becomes False)

How it works: Condition is checked before each iteration. As soon as it's False, the loop stops. Miss the increment/decrement → infinite loop (a real production hazard — can hang a script or burn CPU).

`while` — Practical Patterns

# Countdown (decrement)
a = 10
while a > 0:
    print('Devops')
    a -= 1

# Sum of 1 to 20
total = 0
a = 1
while a <= 20:
    total += a
    a += 1
print(total)        # 210

# Product (factorial-style) of 1 to 20
product = 1
a = 1
while a <= 20:
    product *= a
    a += 1
print(product)

# Pattern using while + string repetition
str1 = 'Devops'
i = 0
while i < len(str1):
    print(str1[i] * (i + 1))
    i += 1
# D
# ee
# vvv
# oooo
# ppppp
# ssssss

`for` vs `while` — When to Use Which

Use `for`	Use `while`
You know the iterable / number of repetitions	You don't know how many times — depends on a condition
Looping over list, string, range	Retry logic, polling, waiting for a state

# DevOps: retry logic — classic while True use case
max_attempts = 5
attempt = 0
while attempt < max_attempts:
    print(f'Attempt {attempt+1}: Connecting to server...')
    # if connection succeeds: break
    attempt += 1

`while True` — Infinite Loop Pattern

# Always True — runs forever until break is hit
# Used for: retry logic, polling, menu-driven scripts, password validation

password = 'xyz1234'
p = input('Enter your password: ')
while True:
    if p == password:
        print('Password matched')
        break                          # exits the loop
    else:
        print('Incorrect password. Try again')
        p = input('Enter your password: ')

DevOps real use: polling an AWS resource until it reaches running state, retrying an API call until success, watching a deployment status.

# Simulated polling pattern (real-world shape)
status = 'pending'
checks = 0
while True:
    checks += 1
    print(f'Check #{checks}: status = {status}')
    if status == 'running':
        print('✅ Instance is up')
        break
    if checks >= 5:
        print('❌ Timeout waiting for instance')
        break
    # status = check_instance_status()   # in real code
    status = 'running' if checks == 3 else 'pending'   # simulate

⏭️ Part 2 — Transfer Statements: `break` & `continue`

`break` — Exit the Loop Entirely

lst = [1, 37, 'Python', True, None, 'Devops', 38, 483, 38.438]

# Print elements until None is found, then stop
for i in lst:
    if i == None:
        break
    else:
        print(i)
# Output: 1, 37, Python, True   (stops at None)

`continue` — Skip Current Iteration Only

lst = [1, 37, None, 'Python', True, None, 'Devops', 38, 483, None, 38.438]

# Skip None values, but KEEP looping through the rest
for i in lst:
    if i == None:
        continue          # skip this iteration, move to next
    else:
        print(i)
# Output: 1, 37, Python, True, Devops, 38, 483, 38.438   (all non-None values)

The Key Difference:

Statement	Effect	Loop continues?
`break`	Stops the loop completely	❌ No — exits
`continue`	Skips just this one iteration	✅ Yes — moves to next

`break` in Nested Loops — Only Breaks the Inner Loop

lst = ['peter', 'piper', 'picked']

# break in inner loop only — outer loop continues
for word in lst:
    for char in word:
        print(char)
        break              # breaks INNER loop only — prints first char of each word
# p, p, p   (first letter of each word)

# break in outer loop — stops everything after first word
for word in lst:
    for char in word:
        print(char)
    break                  # breaks OUTER loop — only processes 'peter'
# p, e, t, e, r

# break in both — exits after just ONE character total
for word in lst:
    for char in word:
        print(char)
        break
    break
# p   (only)

⚠️ Critical for interviews: break/continue only affects the loop they're directly inside — not outer loops.

📦 Part 3 — Lists Deep Dive

List Basics

# Heterogeneous — any mix of types allowed
lst = [1, 37, None, 'Python', True, None, 'Devops', 38, 483, None, 38.438]
print(lst, type(lst))

List Operations

# Concatenation — combine two lists
lst = [1, 2, 3, 4, 5]
lst1 = [6, 7, 8, 9, 10]
print(lst + lst1)             # [1,2,3,4,5,6,7,8,9,10]

# ⚠️ + does NOT modify in place — must reassign
lst + [50]
print(lst)                    # unchanged! still [1,2,3,4,5]
lst = lst + [50]              # ✅ now it's updated

# ❌ Can't add a list and a non-iterable directly
# lst + 50   → TypeError: can only concatenate list (not "int") to list

# Repetition
lst = [1, 2, 3, 4, 5]
print(lst * 5)                # repeats the whole list 5 times

# Length
print(len(lst))                # 5

# Indexing
print(lst[2])                  # 3rd element (0-indexed)

# Slicing
print(lst[::1])                 # full copy, same order
print(lst[::-1])                # reversed list

Nested Lists & Deep Indexing

lst = [37, 'Python', 23+37j, True, [1, 36, 'Devops'],
       [['Linux', 'AWS', 'Learnbay'], 73, 73.476, 51, 'Hello'],
       'Sunday', 'weekend',
       ['Hello', ['hey', 'WFH', [37, 26.437, False], 'Functions']]]

print(lst[5][4])      # 'Hello' — index 5 → that sub-list → index 4
print(lst[8][1][2][1])  # drilling 4 levels deep

# DevOps relevance: nested JSON/API responses look exactly like this
# e.g., ec2_response['Reservations'][0]['Instances'][0]['State']['Name']

Mutability — The Core List Property

lst = [1, 2, 3, 4, 5]
print(id(lst))           # e.g. 140234...

lst[2] = 'Python'        # modify in place — item assignment
print(lst)               # [1, 2, 'Python', 4, 5]
print(id(lst))           # SAME id — list was modified, not recreated

Why this matters: Lists are mutable — changing them doesn't create a new object. This is different from strings/tuples (immutable), where any "change" actually creates a brand-new object.

🏗️ Ways to Create a List

# Case 1 — Direct literal (when you know elements)
lst = [1, 2, 34, 5, 6, 'python', True]

# Case 2 — Using list() on an iterable
list('Python')              # ['P','y','t','h','o','n']
list((10, 20, 30, 40))      # from a tuple → [10,20,30,40]
list(range(12, 24))         # from a range → [12,13,...,23]
list(input('Enter: '))      # from input string → char by char list

# ⚠️ list() on a single non-iterable-wrapped value vs a list with one item
list(['Python'])            # ['Python']  — already a list with 1 string
list('Python')              # ['P','y','t','h','o','n']  — string exploded char by char

# Case 3 — List Comprehension (most DevOps-relevant!)
# Syntax: [expression for item in iterable if condition]

lst = [i for i in range(1, 11)]                     # [1,2,...,10]
lst = [i**2 for i in range(1, 21) if i % 2 == 0]    # squares of even numbers 1-20

# Compare to the manual for-loop equivalent:
lst = []
for i in range(1, 11):
    lst += [i]
# vs the one-liner:
lst = [i for i in range(1, 11)]

🛠️ List Methods — Full Reference

dir(lst)    # shows all available methods on a list object

`append()` — Add ONE Item to the End

lst = [1, 2, 3, 4, 5]
lst.append(30)              # [1,2,3,4,5,30]
lst.append([30, 50])        # adds the WHOLE list as ONE nested item
                             # [1,2,3,4,5,30,[30,50]]  ⚠️ not flattened!
lst.append('Python')        # adds as a single string element

# ❌ append() only takes ONE argument
# lst.append(30, 50)   → TypeError

`extend()` — Add Items One-by-One (Flatten an Iterable In)

lst = [100, 200, 300, 400, 500]
lst.extend('Python')        # adds each CHARACTER separately
print(lst)                  # [100,200,300,400,500,'P','y','t','h','o','n']

lst = [1, 2, 3, 4, 5]
lst.extend([30, 50])        # adds each element separately
print(lst)                  # [1,2,3,4,5,30,50]

append() vs extend() — THE classic interview question:

Method	Adds	Example
`append(x)`	`x` as a single element (even if `x` is a list)	`lst.append([1,2])` → `[...,[1,2]]`
`extend(x)`	each element of `x` individually	`lst.extend([1,2])` → `[...,1,2]`

`insert()` — Insert at a Specific Index

# lst.insert(index, value)
lst = [1, 37, None, 'Python', True]
lst.insert(2, 'Sunday')      # inserts BEFORE index 2, shifts rest right
print(lst)                   # [1, 37, 'Sunday', None, 'Python', True]

lst.insert(-2, 'Python')     # works with negative indexing too

`count()` — Count Occurrences

lst = [1, 2, 3, 1, 2, 3, 1, 2, 3]
print(lst.count(2))          # 3
print(lst.count(1))          # 3

`index()` — Find Position of First Occurrence

lst = [1, 2, 3, 1, 2, 3]
print(lst.index(2))          # 1 (first occurrence, left to right)
print(lst.index(20))         # ❌ ValueError: 20 is not in list

`pop()` — Remove by Index (Returns the Removed Value)

# lst.pop(index)  — default removes the LAST item
lst = [100, 200, 300, 400, 500]
removed = lst.pop(2)          # removes index 2 → returns 300
print(removed)                # 300
print(lst)                    # [100,200,400,500]

lst.pop()                     # removes LAST item by default
lst.pop(-2)                   # negative indexing works too

# ❌ Invalid index → IndexError: pop index out of range

`remove()` — Remove by VALUE (Not Index)

# lst.remove(value) — removes FIRST occurrence, returns nothing
lst = [100, 200, 300, 400, 300, 500]
lst.remove(300)               # removes the FIRST 300 only
print(lst)                    # [100,200,400,300,500]

# ❌ Value not in list → ValueError

pop() vs remove() — Interview Favourite:

Method	Removes by	Returns	Error if missing
`pop(index)`	position/index	the removed value	`IndexError`
`remove(value)`	the value itself	`None`	`ValueError`

`reverse()` — Reverses In Place

lst = [1, 2, 3, 4, 5]
lst.reverse()
print(lst)        # [5,4,3,2,1] — modifies original list

`sort()` — Sorts In Place (Ascending by Default)

lst = [72, 48, 216, 4873, 462, 9]
lst.sort()                    # ascending
lst.sort(reverse=True)        # descending

# Works on strings too (alphabetical / ASCII order)
lst = list('coiuhihcbciygdhs')
lst.sort()

# ⚠️ sort() FAILS on mixed/heterogeneous types
lst = [1, 'Python', True, 38.438]
# lst.sort()   → TypeError: can't compare str and int

`clear()` — Empty the List (Same Object, Same `id()`)

lst = [1, 37, 'Python', True]
print(id(lst))
lst.clear()
print(lst)            # []
print(id(lst))        # SAME id — emptied in place, not a new object

`copy()` — Create an Independent Copy (CRITICAL CONCEPT)

# ❌ WITHOUT copy() — both variables point to the SAME list (same id)
lst = [1, 2, 3, 4, 5]
lst1 = lst                    # NOT a copy — just another name for the same object
lst1[2] = 93
print(lst1)                   # [1,2,93,4,5]
print(lst)                    # [1,2,93,4,5]  ⚠️ original also changed!

# ✅ WITH copy() — independent list, different id
lst = [1, 2, 3, 4, 5]
lst1 = lst.copy()             # new object, different id
lst1[2] = 93
print(lst1)                   # [1,2,93,4,5]
print(lst)                    # [1,2,3,4,5]  ✅ original untouched

This is one of the most important bugs to understand in Python. lst1 = lst does NOT copy — it just creates a second reference to the exact same list in memory.

Manual String Split (Building Towards `.split()`)

# Class built this manually to understand the LOGIC behind str.split()
str1 = 'Peter piper picked a pack of pickeled peppers'

lst = []
res = ''
for char in str1:
    if char != ' ':
        res += char
    else:
        lst.append(res)
        res = ''
lst.append(res)          # don't forget the last word!
print(lst)
# ['Peter', 'piper', 'picked', 'a', 'pack', 'of', 'pickeled', 'peppers']

# This is literally what str.split() does internally:
print(str1.split())      # same result, one line

☁️ DevOps / Cloud Use Cases

# 1. Poll AWS instance status until running (while True pattern)
status = 'pending'
attempts = 0
while True:
    attempts += 1
    if status == 'running':
        print('✅ Instance ready')
        break
    if attempts > 10:
        print('❌ Timeout — instance not ready')
        break
    # status = get_instance_status()   # real AWS SDK call

# 2. Retry failed deployment with break on success
max_retries = 3
attempt = 0
while attempt < max_retries:
    attempt += 1
    print(f'Deploy attempt {attempt}...')
    success = (attempt == 2)        # simulated
    if success:
        print('✅ Deployed')
        break
else:
    print('❌ All retries failed')

# 3. Filter out failed health checks using continue
servers_status = ['ok', 'fail', 'ok', None, 'ok', 'fail']
for status in servers_status:
    if status is None or status == 'fail':
        continue
    print(f'✅ Server status: {status}')

# 4. Build list of running instance IDs with list comprehension
instances = [
    {'id': 'i-001', 'state': 'running'},
    {'id': 'i-002', 'state': 'stopped'},
    {'id': 'i-003', 'state': 'running'},
]
running_ids = [inst['id'] for inst in instances if inst['state'] == 'running']
print(running_ids)        # ['i-001', 'i-003']

# 5. extend() to merge server lists from multiple regions
us_servers = ['web-us-1', 'web-us-2']
eu_servers = ['web-eu-1', 'web-eu-2']
all_servers = []
all_servers.extend(us_servers)
all_servers.extend(eu_servers)
print(all_servers)

# 6. copy() before modifying a shared config list
base_ports = [22, 80, 443]
web_ports = base_ports.copy()
web_ports.append(8080)
print(base_ports)         # unaffected — still [22, 80, 443]
print(web_ports)          # [22, 80, 443, 8080]

❌ Common Mistakes

Mistake	Code	Fix
Forgetting increment in `while`	`while a<=10: print(a)` (no `a+=1`)	Infinite loop — always increment/decrement
`lst + [x]` doesn't modify list	`lst + [50]` then expects `lst` changed	Reassign: `lst = lst + [50]`
`append()` vs `extend()` confusion	`lst.append([1,2])` → adds nested list, not 2 items	Use `extend()` to flatten in
`lst1 = lst` thinking it's a copy	Modifying `lst1` also changes `lst`	Use `lst1 = lst.copy()`
`sort()` on mixed types	`[1,'a',True].sort()`	TypeError — sort only same-type lists
`pop()` on empty/invalid index	`[].pop()`	IndexError — check length first
`remove()` on missing value	`lst.remove(999)` when not present	ValueError — check `in` first
`break` only exits ONE loop level	Expecting `break` to exit nested loops fully	Need a flag or function return for full exit

🎯 Interview Points

"Difference between append() and extend()?"
→ append() adds its argument as a single element (even a whole list becomes one nested item). extend() iterates through its argument and adds each element individually.
"Difference between break and continue?"
→ break exits the loop entirely. continue skips only the current iteration and proceeds to the next one.
"Difference between pop() and remove()?"
→ pop(index) removes by position and returns the value; default removes the last item. remove(value) removes the first matching value and returns nothing.
"What happens when you do lst1 = lst vs lst1 = lst.copy()?"
→ lst1 = lst makes lst1 point to the SAME object — changes to one affect the other. .copy() creates an independent object with a new id().
"Why are lists mutable?"
→ Their contents can be changed after creation (id() stays the same) — unlike strings/tuples where any change creates a new object.
"What's a list comprehension and why use it?"
→ A concise one-line way to build a list: [expr for item in iterable if condition]. More Pythonic and often faster than a manual for-loop + append.
"When would you use while True over a regular while?"
→ When the exit condition is checked INSIDE the loop body rather than at the top — e.g., retry logic, polling, "ask again until valid input."

📚 Knowledge Base — Quick Revision

# ── WHILE LOOP ──────────────────────────────────────────────
a = 1                       # 1. init
while a <= 10:               # 2. condition
    ...
    a += 1                    # 3. increment — NEVER forget this

while True:                  # infinite — needs break
    if condition:
        break

# ── BREAK / CONTINUE ────────────────────────────────────────
break        # exits the loop entirely
continue     # skips current iteration, loop continues

# ── LIST CREATION ───────────────────────────────────────────
lst = [1, 2, 3]                       # literal
lst = list('abc')                     # from string → ['a','b','c']
lst = list(range(5))                  # from range
lst = [i for i in range(10)]          # comprehension
lst = [i**2 for i in range(10) if i%2==0]   # comprehension + filter

# ── LIST OPERATIONS ─────────────────────────────────────────
lst + lst2          # concatenation (new list, must reassign)
lst * 3              # repetition
len(lst)              # length
lst[i]                # indexing
lst[::-1]             # reverse via slicing

# ── LIST METHODS (mutating, in-place) ───────────────────────
lst.append(x)         # add ONE item to end
lst.extend(iterable)  # add items individually
lst.insert(i, x)      # insert before index i
lst.pop(i)            # remove by index, RETURNS value (default: last)
lst.remove(x)         # remove by VALUE, first match, returns None
lst.reverse()         # reverse in place
lst.sort()            # ascending; sort(reverse=True) for descending
lst.clear()           # empty the list (same id)
lst.count(x)          # occurrences of x
lst.index(x)          # first index of x (ValueError if missing)
lst.copy()            # independent shallow copy (new id)

# ── CRITICAL GOTCHA ─────────────────────────────────────────
lst1 = lst             # ❌ same object — NOT a copy
lst1 = lst.copy()      # ✅ independent copy

🏋️ Practice Questions

Easy

Write a while loop that prints numbers from 10 down to 1.
Given lst = [10, 20, 30, 40, 50], use append() to add 60, then extend() to add [70, 80]. Print the list after each step.
Use a list comprehension to create a list of cubes (x³) for numbers 1 to 10.

Medium

Write a while True loop that keeps asking the user to enter a number until they enter a number greater than 100, then breaks and prints "Valid number entered."
Given lst = [5, 10, None, 15, None, 20], write a loop using continue to print only non-None values, and a separate loop using break to print values until the first None.
Demonstrate the lst1 = lst vs lst1 = lst.copy() bug with code: create a list, assign it without .copy(), modify the copy, and show the original also changed. Then fix it using .copy().

DevOps-Focused

Retry Connector: Write a while loop that simulates connecting to a server. It should try up to 5 times. Use a variable connected = False that becomes True on the 3rd attempt (simulated). Print each attempt, and break with a success message once connected, or a failure message after 5 failed attempts.
Server List Builder: You have two lists: prod_servers = ['web-01','web-02'] and staging_servers = ['stg-01','stg-02']. Use extend() to build one combined all_servers list. Then use a list comprehension to create prod_only containing only servers that start with 'web'. Print both results.

AWS IAM Deep Dive — Roles, Policies, Groups, EC2 & Everything in Between

Tejas Shinkar — Sat, 20 Jun 2026 04:02:06 +0000

🔐 AWS IAM Deep Dive

Authentication · Authorization · Accounting · Roles · EC2 Basics

This is part of my ongoing AWS learning journey as I transition from Systems Engineer to Cloud/DevOps. I'm documenting everything I learn — concepts, hands-on steps, and the "why" behind each one.

Why IAM Exists — The Real Answer

When you first create an AWS account, you get a Root user. It has zero restrictions — it can delete everything, rack up unlimited bills, create or destroy any resource. That's exactly why you should almost never use it.

Real teams have multiple people — developers, DevOps engineers, auditors, managers. You don't want everyone logging into the same all-powerful Root account. You need controlled, tracked, individual access. That's what IAM solves.

IAM answers one fundamental question: Who can do what, on which AWS resource — and is it being recorded?

That question maps perfectly to a security framework called AAA.

The AAA Framework

Every access control system — not just AWS — is built on three pillars:

Pillar	Question it answers	How AWS implements it
Authentication	Who are you?	Credentials (username + password) + MFA
Authorization	What can you do?	IAM Policies and Roles
Accounting	What did you do?	AWS CloudTrail (full audit logs)

When you log into AWS and try to start an EC2 instance, all three happen in sequence — AWS verifies your identity, checks if you're allowed to start EC2, and logs that you did it. Every single time.

MFA — The Second Lock

A password alone isn't enough. If someone gets your password, they're in. MFA adds a second factor — a time-based OTP from your phone — so even a stolen password is useless without the physical device.

Three MFA types:

Type	How it works	When to use
Virtual MFA	Google Authenticator / Authy app	✅ Default choice for learning
Hardware MFA	Physical YubiKey device	✅✅ Best for Root account
SMS/Voice	OTP via text message	⚠️ Weakest, avoid if possible

Always enable MFA on Root first. Then on your IAM user too.

IAM Users

An IAM User is an identity created inside your AWS account for a specific person or application. Unlike Root, it starts with zero permissions — it can log in but can't do anything until you attach a policy.

Creating an IAM User (what you actually did):

IAM Console → Users → Create User
→ Set username
→ Enable Console access + set password
→ (Optional) Add tags
→ Attach policy directly OR add to a group
→ Done — user can now log in with limited access

One important thing: an IAM user with permission to create users can create more IAM users. Permissions propagate — so be careful what you give.

IAM Groups — Managing at Scale

Imagine you have 10 developers. Attaching policies to each one individually is painful and error-prone. Groups solve this.

A Group is a collection of IAM users. Attach a policy to the group once — every user in it inherits those permissions automatically.

Group: DevOpsTeam  →  Policy: PowerUserAccess
   ├── User: tejas        → gets PowerUserAccess ✅
   ├── User: ravi         → gets PowerUserAccess ✅
   └── User: priya        → gets PowerUserAccess ✅

Add a new team member → add to group → done.
Remove someone → remove from group → done.

Key rules:

One user can belong to multiple groups
Groups cannot contain other groups (no nesting)
Default quota: 300 groups per account

Creating via Group (the cleaner approach):

Create IAM Group → Attach policy to group
→ Create IAM User → Add user to group
→ User inherits group's permissions automatically

IAM Policies — The Permission Document

A Policy is a JSON document that defines exactly what is allowed or denied. Every time you make a request in AWS — clicking a button, running a CLI command, an app calling an API — AWS checks the relevant policies to decide: allow or deny?

Basic structure:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

Field	What it means
`Effect`	`Allow` or `Deny`
`Action`	Which API operation (e.g. `s3:GetObject`, `ec2:*`)
`Resource`	Which specific resource (by ARN)
`Condition`	Optional — e.g. "only if MFA is active"

The one rule that matters most:

Explicit Deny > Everything else
No Policy = Denied by default
Explicit Allow = Access granted (unless a Deny exists)

If a Deny exists anywhere in any applicable policy, it always wins — even if an Allow exists somewhere else.

What about combined permissions?
AWS combines all Allow permissions across all sources:

User has: AmazonEC2FullAccess (direct policy)
Group has: ReadOnlyAccess
Result:    EC2 Full Access + ReadOnly on everything else
           (the more permissive wins, unless there's an explicit Deny)

SCP — The Ceiling Above IAM

You now understand IAM policies. But there's something that sits above IAM entirely.

SCP = Service Control Policy — it's part of AWS Organizations (used when you manage multiple AWS accounts together).

The key distinction:

	IAM Policy	SCP
Attached to	Users, Groups, Roles	AWS Accounts / Organizational Units
Does it grant permissions?	✅ Yes	❌ No — only limits
Can it restrict Root?	❌ No	✅ Yes
Who uses it?	Everyone	Enterprises with multiple accounts

SCP defines the maximum permissions possible in an account. IAM policies still need to grant the actual access — but they can never exceed what SCP allows.

Example: SCP says "no S3 access allowed in this account." Even if an IAM policy gives a user S3FullAccess, they still can't access S3. SCP wins.

For now, SCP is mostly relevant at enterprise scale. Keep it in mind — it'll come up in interviews.

ARN — Every Resource Has a Unique ID

ARN = Amazon Resource Name. Every single thing in AWS — users, roles, buckets, instances — has a globally unique ARN.

Format:

arn:aws:SERVICE:REGION:ACCOUNT-ID:RESOURCE

Examples:

arn:aws:iam::123456789012:user/tejas           → IAM User
arn:aws:iam::123456789012:role/S3AccessRole    → IAM Role
arn:aws:s3:::my-bucket                          → S3 Bucket
arn:aws:ec2:ap-south-1:123456789:instance/i-abc → EC2 Instance

ARNs are used everywhere — in trust policies, resource-based policies, CloudTrail logs. When you're adding a user to a role's trust policy, you paste their ARN.

IAM Roles — Temporary Identity, No Password

This is where most beginners get confused. Let's clear it up completely.

An IAM Role is a temporary identity with permissions — no username, no password. It's meant to be assumed — picked up temporarily by a user or an AWS service, used, and then dropped.

The visitor badge analogy:
Think of a role as a visitor badge at an office. A visitor picks up the badge at reception (assumes the role), gets access to specific floors (permissions), and returns the badge when done (session ends). No permanent access. No long-term credentials.

IAM User vs IAM Role

	IAM User	IAM Role
Has password?	✅ Yes	❌ No
Permanent?	✅ Yes	❌ Temporary (assumed)
For?	Human beings	Services or temporary access
Credentials	Long-term	Short-term, auto-rotated

Trust Policy — Who Can Assume a Role?

Every IAM Role has two things attached:

Permission Policy — what the role is allowed to do
Trust Policy — who is allowed to assume this role

The Trust Policy is a JSON document that answers: "Who gets to pick up this badge?"

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::123456789012:user/tejas"
    },
    "Action": "sts:AssumeRole"
  }]
}

This says: only the IAM user tejas can assume this role.

Role Switching:

Here's the complete flow:

Step 1: Create a Role
  IAM → Roles → Create Role
  → Attach permission: AmazonS3FullAccess
  → Name it: S3AccessRole

Step 2: By default, nobody can assume it
  Your IAM user tries to switch → fails ❌

Step 3: Edit the Trust Policy
  Open S3AccessRole → Trust Relationships → Edit
  → Add your IAM user's ARN as a trusted principal
  → Save

Step 4: Switch Role (from Console top-right)
  Click your username → Switch Role
  → Enter Account ID + Role name
  → You're now operating AS the role

Step 5: What changes?
  ✅ You can now access S3
  ❌ Your original EC2 permissions are SUSPENDED
  → Role permissions fully replace your user permissions for this session

Step 6: Switch back
  Click role name (top-right) → Switch Back
  → Back to your original IAM user permissions
  → S3 access gone

That EC2 access disappearing when you switched to S3 role wasn't a bug — that's exactly how role switching works. The role's permissions replace your current ones for the duration of the session.

Service Principal — Roles for AWS Services

So far, roles have been assumed by IAM users (humans). But AWS services need to access other services too — and they use roles for that.

A Service Principal is the identity of an AWS service, used in trust policies:

ec2.amazonaws.com      → EC2 can assume roles
lambda.amazonaws.com   → Lambda can assume roles
s3.amazonaws.com       → S3 can assume roles

Example: EC2 accessing S3 automatically

You want your EC2 instance to read/write S3 without hardcoding any credentials in your code.

Bad approach ❌:
  aws_access_key_id = 'AKIAIOSFODNN7EXAMPLE'  ← hardcoded in code
  aws_secret_access_key = 'wJalrXUt...'        ← if pushed to GitHub, you're done

Right approach ✅:
  Create a Role with S3 permissions
  Trust Policy: ec2.amazonaws.com can assume this role
  Attach role to EC2 via Instance Profile
  EC2 automatically gets temporary, rotating credentials
  Your code just calls S3 — no credentials anywhere

IAM Instance Profile — The Bridge Between EC2 and IAM

An Instance Profile is a container that wraps an IAM Role and delivers it to an EC2 instance. When you attach a role to EC2 via the Console, an Instance Profile is created automatically behind the scenes.

Complete flow:

IAM Role (S3AccessRole)
    ↓ wrapped in
Instance Profile
    ↓ attached to
EC2 Instance
    ↓ app calls
AWS Metadata Service (169.254.169.254) — internal AWS endpoint
    ↓ returns
Temporary credentials (auto-rotated every hour)
    ↓ used to
Access S3 ✅ — no hardcoded keys, no manual rotation

Lab steps:

Step 1: Create Role
  IAM → Roles → Create Role
  Trusted entity: AWS Service → EC2
  Attach policy: AmazonS3FullAccess
  Name: EC2-S3-Role

Step 2: Attach to EC2
  EC2 Console → Select Instance
  Actions → Security → Modify IAM Role
  Select: EC2-S3-Role → Update

Step 3: Test
  SSH into EC2 → run: aws s3 ls
  → Lists your S3 buckets ✅ (zero credentials configured)

EC2 Basics — Launching Your First Instance

EC2 = Elastic Compute Cloud — a virtual machine running on AWS hardware. When you launch one, you're renting a slice of a physical server in an AWS data center.

The Key Components

AMI (Amazon Machine Image)
A pre-built template containing the OS, software, and configuration. It's the blueprint your EC2 instance is built from.

Amazon Linux 2, Ubuntu 22.04, Windows Server — these are all AMIs
AMIs are Region-specific — if you want the same AMI in another Region, you must manually copy it. AWS does not auto-replicate AMIs.

Instance Type
Defines CPU, RAM, and network capacity.

t2.micro — 1 vCPU, 1 GB RAM — Free Tier eligible ✅
Naming: t3.medium → t = family, 3 = generation, medium = size

Key Pair
Used to SSH into your EC2 instance securely — no password needed.

AWS puts the public key on your EC2 instance
You download the private key (.pem file)
You use .pem to prove your identity: ssh -i key.pem ec2-user@<ip>
Download once. If you lose it, you lose SSH access. No recovery.

Security Group
Acts as a virtual firewall. Controls what traffic can reach your instance.

Port 22 (SSH) — for terminal access
Port 80 (HTTP) — for web traffic
Port 443 (HTTPS) — for secure web traffic

User Data
A script that runs once automatically the first time EC2 boots. Used to install software, configure servers, deploy apps — all without manual steps.

#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo "<h1>Hello from EC2!</h1>" > /var/www/html/index.html

This script installs Apache web server and creates a basic webpage — automatically on first boot.

EC2 Launch Flow (What You Did)

EC2 Console → Launch Instance
  ↓
Name/Tags → give it a meaningful name
  ↓
AMI → Amazon Linux 2 (Free Tier)
  ↓
Instance Type → t2.micro (Free Tier)
  ↓
Key Pair → Create new → Download .pem file (save it!)
  ↓
Security Group → Allow SSH (22) + HTTP (80)
  ↓
Storage → Default 8 GB (Free Tier covers this)
  ↓
Advanced → User Data → paste startup script
  ↓
Launch Instance
  ↓
Wait ~2 minutes → Instance state: Running
  ↓
Copy Public IP → open in browser → "Hello from EC2!" ✅

Two Policy Types — Identity vs Resource Based

You've now seen policies attached to users, groups, and roles. But policies can also be attached to resources directly.

Identity-Based Policy

Attached to: IAM user, group, or role
Says: "This identity can perform these actions"
No Principal field needed (the identity it's attached to is the principal)
Example: AmazonEC2FullAccess attached to your IAM user

Resource-Based Policy

Attached to: a resource (S3 bucket, SQS queue, etc.)
Says: "These identities are allowed to access me"
Has a Principal field — explicitly names who can access
Example: S3 bucket policy allowing a specific IAM user to read objects

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::123456789:user/tejas"
    },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-bucket/*"
  }]
}

Quick Revision

AAA FRAMEWORK
  Authentication  → Who are you? (credentials + MFA)
  Authorization   → What can you do? (IAM policies)
  Accounting      → What did you do? (CloudTrail)

PERMISSION HIERARCHY
  SCP (Organizations) → ceiling, limits max permissions
  IAM Policies       → actual grants within that ceiling
  Explicit Deny      → always wins over everything

IAM BUILDING BLOCKS
  User    → permanent identity, has password, for humans
  Group   → collection of users, attach policy once
  Role    → temporary identity, no password, assumed
  Policy  → JSON doc defining Allow/Deny on resources
  ARN     → unique ID for every AWS resource

ROLE SWITCHING
  Add user ARN to role's Trust Policy
  User switches role → gets role permissions temporarily
  Original permissions suspended during session
  Switch back → original permissions restored

INSTANCE PROFILE (EC2 → S3)
  Create Role (trust: ec2.amazonaws.com) + S3 policy
  Attach to EC2 via Instance Profile
  EC2 auto-gets temp credentials → accesses S3
  Zero hardcoded credentials ✅

EC2 BASICS
  AMI      = OS template (Region-specific, must copy to use elsewhere)
  Key Pair = SSH access (.pem — download once, don't lose)
  User Data = startup script (runs once on first boot)
  t2.micro = Free Tier instance type

Interview Questions

Q1: What is the difference between an IAM User and an IAM Role?
An IAM User has permanent credentials (username + password) and is meant for human beings. An IAM Role has no password — it provides temporary credentials and is assumed by users or AWS services when needed. Roles are best practice for service-to-service communication.

Q2: What is a Trust Policy and why is it needed?
A Trust Policy is a JSON document attached to an IAM Role that defines who is allowed to assume that role. Without it, nobody can use the role — not even admins. You explicitly add trusted principals (IAM user ARNs or service principals like ec2.amazonaws.com).

Q3: What happens when you switch roles in AWS?
Your current IAM user permissions are temporarily suspended and replaced by the role's permissions for that session. When you switch back, your original permissions are restored. This is why EC2 access disappeared when you switched to the S3 role.

Q4: What is an Instance Profile and why use it instead of access keys?
An Instance Profile is a container that delivers an IAM Role to an EC2 instance. It gives EC2 automatic, temporary, rotating credentials via the metadata service — no hardcoded keys needed. This is far more secure than embedding access keys in code, which can be accidentally exposed.

Q5: What is the difference between SCP and IAM policies?
IAM policies grant permissions to users, groups, or roles within an account. SCPs (Service Control Policies) sit above IAM at the Organizations level and define the maximum permissions allowed in an account — they restrict, not grant. Even the Root user is subject to SCP.

Q6: An IAM user has EC2FullAccess via direct policy and ReadOnlyAccess via their group. What's their effective permission?
They get EC2 Full Access plus ReadOnly on everything else. AWS combines all Allow permissions. However, if an Explicit Deny exists anywhere, that overrides all Allows.

Practice Tasks

Create an IAM Group called DevOpsTeam, attach PowerUserAccess, create a user and add them to the group. Log in as that user and verify you can access EC2 but not billing.
Create an IAM Role with AmazonS3ReadOnlyAccess. Edit its Trust Policy to add your IAM user's ARN. Switch to that role and verify S3 access. Notice EC2 access is gone.
Launch an EC2 instance with a User Data script that installs Apache. Access the public IP in your browser and confirm the webpage loads.
Create an IAM Role for EC2 with S3 access. Attach it to a running EC2 instance via Modify IAM Role. SSH in and run aws s3 ls — confirm it works without any credentials configured.
Open AmazonEC2FullAccess policy JSON in the IAM Console. Find what ec2:* means and identify 3 specific actions it includes.

Part of my AWS Cloud + DevOps learning journey — documenting everything as I transition from Systems Engineer to Cloud/DevOps Engineer.

String Methods, Conditional Statements (if/elif/else), Loops (for, range())

Tejas Shinkar — Fri, 19 Jun 2026 09:42:13 +0000

🐍 Python for DevOps — Class 5 Notes

Topic: String Methods, Conditional Statements (`if/elif/else`), Loops (`for`, `range()`)

📌 Key Concepts Overview

Concept	One-Line Definition
String Methods	Built-in functions to inspect and transform strings
`if / elif / else`	Execute code blocks based on conditions
`for` loop	Iterate over any iterable — string, list, range, dict
`range()`	Generate a sequence of numbers to loop over
Nested `if`	An `if` block inside another `if` block
Nested `for`	A `for` loop inside another `for` loop

🔤 Part 1 — String Methods (Continuation from Class 4)

Checking String Case

# islower() — True if ALL characters are lowercase
'devops'.islower()     # True
'Devops'.islower()     # False

# isupper() — True if ALL characters are uppercase
'DEVOPS'.isupper()     # True
'Devops'.isupper()     # False

# Practical: normalize before comparing
env = input('Enter environment: ')
if env.lower() == 'production':
    print('⚠️ Deploying to PROD!')

Checking Digit / Numeric Strings

Method	Accepts	Rejects	DevOps Use
`isdecimal()`	`0-9` only	decimals, superscripts	Port validation
`isdigit()`	`0-9` + superscripts	decimal points	Version numbers
`isnumeric()`	`0-9` + superscripts + fractions	decimal points	Broadest check

# Key difference — decimal point breaks all three
'8080'.isdecimal()    # True
'8080'.isdigit()      # True
'8080'.isnumeric()    # True

'80.80'.isdecimal()   # False — dot is not a digit
'80.80'.isdigit()     # False
'80.80'.isnumeric()   # False

# DevOps use: validate user input before casting
port_input = input('Enter port: ')
if port_input.isdecimal():
    port = int(port_input)
else:
    print('❌ Invalid port — must be a whole number')

`replace()` — Find and Replace in Strings

# str.replace('old', 'new')
'Python'.replace('P', 'J')              # 'Jython'
'prod-server-01'.replace('-', '_')      # 'prod_server_01'
'This is python class'.replace('i','a') # 'Thas as python class' — ALL occurrences

# DevOps: sanitize strings for use in AWS resource names (no spaces allowed)
resource_name = 'my resource name'
safe_name = resource_name.replace(' ', '-')  # 'my-resource-name'

# Remove spaces entirely
'Pyt hon'.replace(' ', '')    # 'Python'

`startswith()` / `endswith()`

log_line = 'ERROR: Connection refused on port 5432'

# Route logs based on prefix
if log_line.startswith('ERROR'):
    print('🚨 Alert triggered')
elif log_line.startswith('WARN'):
    print('⚠️ Warning logged')

# Check file types
filename = 'deploy.sh'
if filename.endswith('.sh'):
    print('Shell script — set execute permission')
elif filename.endswith('.py'):
    print('Python script')
elif filename.endswith('.tf'):
    print('Terraform file')

🔀 Part 2 — Conditional Statements

`if` — Single Condition

# if condition:
#     block runs only when condition is True
# (anything outside the block always runs)

a = 392
if a % 2 == 0:
    print('Even')        # runs only if condition True

print('Done')            # ALWAYS runs — outside if block

# ⚠️ Common mistake from class:
# Putting "Number is odd" OUTSIDE the if block
# It prints even when number is even — logic bug

`if-else` — Two Branches

# if condition:
#     runs when True
# else:
#     runs when False

a = int(input('Enter a number: '))
if a % 2 == 0:
    print('Even')
else:
    print('Odd')

# DevOps: health check routing
status_code = 200
if status_code == 200:
    print('✅ Service healthy')
else:
    print(f'❌ Service unhealthy — code: {status_code}')

Multiple `if` vs `if-elif-else` — Critical Difference

# ❌ Multiple if — checks ALL conditions independently (inefficient)
a = 1
if a == 1: print('One')      # matches — prints One
if a == 2: print('Two')      # checks anyway — no match
if a == 3: print('Three')    # checks anyway — no match
# All 5 conditions are evaluated even after a match

# ✅ if-elif-else — stops at FIRST match (efficient)
if a == 1:
    print('One')             # matches — STOPS here
elif a == 2:
    print('Two')             # SKIPPED
elif a == 3:
    print('Three')           # SKIPPED
else:
    print('Out of range')    # SKIPPED

Rule of thumb: Use elif when conditions are mutually exclusive (only one can be true). Use multiple if when you need to check all conditions independently.

`if-elif-else` — Full Pattern

# HTTP status code handler (real DevOps pattern)
status = int(input('Enter HTTP status code: '))

if status == 200:
    print('OK — Success')
elif status == 201:
    print('Created — Resource provisioned')
elif status == 400:
    print('Bad Request — Check input params')
elif status == 401:
    print('Unauthorized — Check IAM/credentials')
elif status == 403:
    print('Forbidden — Check permissions')
elif status == 404:
    print('Not Found — Resource missing')
elif status == 500:
    print('Internal Server Error — Check app logs')
else:
    print(f'Unknown status: {status}')

`in` Operator with Conditionals

# Check membership — works on strings, lists, tuples, sets, dicts
x = input('Enter a letter: ')

if x.lower() in 'aeiou':
    print('Vowel')
else:
    print('Consonant')

# DevOps: check if region is valid
region = input('Enter AWS region: ')
valid_regions = ['ap-south-1', 'us-east-1', 'eu-west-1']
if region in valid_regions:
    print('✅ Valid region')
else:
    print('❌ Invalid region')

Nested `if-else`

# When you need to check conditions inside conditions
# Class used: pass Python AND Linux AND AWS to clear exam

p = int(input('Python score: '))
l = int(input('Linux score: '))
a = int(input('AWS score: '))

if p > 75:
    if l > 75:
        if a > 75:
            print('✅ Cleared all subjects')
        else:
            print('❌ Work hard in AWS')
    else:
        print('❌ Work hard in Linux')
else:
    print('❌ Work hard in Python')

# DevOps equivalent: multi-condition deployment gate
cpu = 45
memory = 70
disk = 60

if cpu < 80:
    if memory < 85:
        if disk < 90:
            print('✅ All checks passed — safe to deploy')
        else:
            print('❌ Disk usage too high')
    else:
        print('❌ Memory usage too high')
else:
    print('❌ CPU usage too high')

🔁 Part 3 — Loops

`for` Loop — Iterating Over Iterables

# Syntax: for variable in iterable:
#             block

# Iterables: string, list, tuple, set, dict, range, bytes

# Over a string — character by character
for char in 'Python':
    print(char)     # P y t h o n (each on new line)

# Over a list
servers = ['web-01', 'db-01', 'cache-01']
for server in servers:
    print(f'Pinging {server}...')

# Over a list with condition
for server in servers:
    if server.startswith('db'):
        print(f'⚠️ Database server: {server}')

`range()` — Generating Number Sequences

# range(stop)          → 0 to stop-1
# range(start, stop)   → start to stop-1
# range(start, stop, step) → with step size

range(10)           # 0, 1, 2, ..., 9
range(1, 11)        # 1, 2, ..., 10
range(0, 51, 2)     # 0, 2, 4, ..., 50  (even numbers)
range(-1, -7, -1)   # -1, -2, ..., -6  (reverse traversal)

# Common patterns
for i in range(10):
    print('DevOps')     # print 10 times

for i in range(1, 51):
    if i % 2 == 0:
        print(i)        # even numbers 2-50

Looping with Index — `range(len())`

# When you need BOTH the index and the value
str1 = 'Python'
for i in range(len(str1)):
    print(i, str1[i])
# 0 P
# 1 y
# 2 t
# 3 h
# 4 o
# 5 n

# Find all positions of a character
log = 'ERROR: disk full. ERROR: retry failed.'
for i in range(len(log)):
    if log[i] == 'E' and log[i:i+5] == 'ERROR':
        print(f'ERROR found at index {i}')

String + Loop Patterns

# Extract vowels from a string
str1 = 'wqciecneucqbcjeqbviuqciwqnkeqnv'
for char in str1:
    if char.upper() in 'AEIOU':
        print(char)

# Get all uppercase letters
str1 = 'This is Python Class.'
for char in str1:
    if char.isupper():
        print(char)    # T, P, C

# Reverse a string using loop
str1 = 'Python'
rev_str = ''
for i in range(-1, -len(str1)-1, -1):
    rev_str += str1[i]
print(rev_str)         # nohtyP

Nested `for` Loops

# Loop inside a loop — inner loop completes FULLY for each outer iteration

# Class example: print each word AND each character
lst = ['Peter', 'piper', 'picked']

for word in lst:
    print(word)
    for char in word:
        print(char)

# DevOps: scan all servers for all ports
servers = ['web-01', 'db-01']
ports = [22, 80, 443]

for server in servers:
    for port in ports:
        print(f'Checking {server}:{port}')
# web-01:22, web-01:80, web-01:443, db-01:22 ...

☁️ DevOps / Cloud Use Cases

# 1. Log line parser — classify log levels
logs = [
    'ERROR: DB connection failed',
    'INFO: Deployment started',
    'WARN: CPU at 82%',
    'ERROR: Pod CrashLoopBackOff',
]

for log in logs:
    if log.startswith('ERROR'):
        print(f'🚨 ALERT: {log}')
    elif log.startswith('WARN'):
        print(f'⚠️  WARN: {log}')
    else:
        print(f'ℹ️  INFO: {log}')

# 2. Port scanner stub (basic)
target_ports = [22, 80, 443, 3306, 5432, 8080, 8443]
for port in target_ports:
    if port in [22, 80, 443]:
        print(f'Port {port} — standard, expected open')
    elif port in [3306, 5432]:
        print(f'Port {port} — DB port, should be private!')
    else:
        print(f'Port {port} — verify if needed')

# 3. Multi-region health check
regions = ['ap-south-1', 'us-east-1', 'eu-west-1']
for region in regions:
    if region.startswith('ap'):
        print(f'{region} → APAC cluster')
    elif region.startswith('us'):
        print(f'{region} → US cluster')
    else:
        print(f'{region} → EU cluster')

# 4. Validate resource name before creating
name = input('Enter resource name: ')
safe_name = name.replace(' ', '-').lower()
if safe_name.replace('-', '').isalnum():
    print(f'✅ Valid name: {safe_name}')
else:
    print('❌ Name has invalid characters')

# 5. Find ERROR count in a log file (simulated)
log_data = 'INFO start\nERROR timeout\nINFO retry\nERROR disk full\nINFO done'
error_count = 0
for line in log_data.split('\n'):
    if line.startswith('ERROR'):
        error_count += 1
print(f'Total errors: {error_count}')

❌ Common Mistakes

Mistake	Code	Fix
Code outside `if` always runs	`print('Odd')` after the `if` block	Put it inside `else`
Multiple `if` instead of `elif`	All conditions check even after match	Use `elif` for exclusive branches
`range(10)` gives 0–9, not 1–10	`range(10)` → 0,1...9	`range(1, 11)` for 1 to 10
Off-by-one in reverse range	`range(-1, -6, -1)` for 6-char string	`range(-1, -len(s)-1, -1)`
`in` on wrong type	`if port in '8080'` → checks substring in string	`if port in [8080, 443]` — use a list
`islower()` on string with spaces	`'hello world'.islower()` → `True`	Spaces are ignored — only letters checked
Forgetting colon after `if`/`for`	`if x == 1` → SyntaxError	Always end with `:`
Wrong indentation	Mixed tabs/spaces	Use 4 spaces consistently

🎯 Interview Points

"Difference between if-elif and multiple if statements?"
→ elif stops after the first match — faster, mutually exclusive. Multiple if evaluates every condition every time.
"What is the difference between isdecimal(), isdigit(), isnumeric()?"
→ isdecimal() is strictest (0-9 only). isdigit() also accepts superscripts. isnumeric() is broadest — accepts fractions too. In DevOps, use isdecimal() for port/number validation.
"How does for loop work internally?"
→ Python calls iter() on the iterable, then repeatedly calls next() until StopIteration. Every for loop is really an iterator under the hood.
"What does range(1, 10) return?"
→ 1 through 9 — stop value is excluded. range(1, 11) for 1 through 10.
"When would you use nested loops in DevOps?"
→ Scanning multiple servers × multiple ports. Processing multi-dimensional data. Parsing nested config structures.
"Can you use in with a dict?"
→ in on a dict checks keys only, not values. 'key' in my_dict → checks keys.
"What is the in operator?"
→ Membership test — returns True if element exists in the iterable. Works on strings, lists, tuples, sets, dict keys.

📚 Knowledge Base — Quick Revision

# ── STRING METHODS ──────────────────────────────────────────
'devops'.islower()              # True
'DEVOPS'.isupper()              # True
'8080'.isdecimal()              # True  ← use for port validation
'8.0'.isdecimal()               # False ← dot breaks it
'hello'.replace('l', 'r')       # 'herro'
'error.log'.endswith('.log')    # True
'ERROR'.startswith('ERR')       # True

# ── CONDITIONALS ────────────────────────────────────────────
if condition:
    ...
elif condition:         # else-if — stops at first match
    ...
else:
    ...

# Use 'in' for membership
if value in [list, of, options]:
    ...

# ── FOR LOOP ────────────────────────────────────────────────
for item in iterable:       # iterates over elements
    ...

for i in range(10):         # 0 to 9
    ...

for i in range(1, 11):      # 1 to 10
    ...

for i in range(0, 10, 2):   # 0,2,4,6,8 (step=2)
    ...

for i in range(len(s)):     # index-based traversal
    print(i, s[i])

# ── NESTED LOOPS ────────────────────────────────────────────
for server in servers:
    for port in ports:
        print(f'{server}:{port}')

# ── STRING REVERSE ───────────────────────────────────────────
rev = ''
for i in range(-1, -len(s)-1, -1):
    rev += s[i]
# or simply: s[::-1]

🏋️ Practice Questions

Easy

Write a script that checks if a given string (like a resource name) contains only lowercase letters using islower(). If not, convert it and print the result.
Use a for loop with range() to print all even numbers from 1 to 100.
Take a filename as input (e.g., deploy.sh, main.tf, config.yml). Use endswith() to print what type of file it is.

Medium

Write a script that takes a port number string as input, validates it using isdecimal(), casts it to int, and checks if it falls in one of these ranges: Well-known (0–1023), Registered (1024–49151), Dynamic (49152–65535). Print the category.
Given a log string like 'INFO start\nERROR timeout\nWARN disk 80%\nERROR crash', loop through each line and count how many are ERROR, WARN, and INFO. Print the counts.
Write a script that builds a "safe" AWS resource name from user input: strip spaces with replace(), convert to lowercase, validate with isdecimal() / isalnum(), and print either the safe name or an error.

DevOps-Focused

Deployment Gate Script: Ask the user for CPU %, memory %, and disk % (all integers). Use nested if-else to decide: if all are under 80%, print "✅ Safe to deploy". If any one exceeds, print which resource is the bottleneck. Use f-strings for output.
Log Analyzer: Given this list of log lines:

   logs = ['ERROR: pod crash', 'INFO: deploy started', 'ERROR: OOM killed',
           'WARN: cpu spike', 'INFO: rollback complete', 'ERROR: disk full']

Loop through and: (a) print ERROR lines in uppercase using .upper(), (b) count total errors, (c) print a final summary: "Found X errors in Y total log lines".

Global Infrastructure + IAM Practical + CloudFront & CDN

Tejas Shinkar — Thu, 18 Jun 2026 08:46:28 +0000

📌 Session Overview

#	Topic
1	AWS Regions — What they are & how isolation works
2	Availability Zones (AZs) — Deep dive
3	Data Centers — The physical foundation
4	Edge Locations & Points of Presence (PoPs)
5	Latency & Fault Tolerance
6	Region Selection Strategy
7	Benefits of AWS Global Infrastructure
8	CloudFront & CDN — Concepts + Demo
9	IAM — Practical Recap & Best Practices
10	Cost Prevention Checklist

🔑 ESSENTIAL AWS TECHNICAL TERMS (Quick Reference)

Before diving in — here are the key terms you'll hear constantly in any AWS/DevOps role:

Term	Simple Meaning
Region	A geographic location with AWS data centers (e.g., Mumbai)
AZ (Availability Zone)	Isolated data center cluster within a Region
Edge Location / PoP	Cache server in a city near end users (for CDN)
CDN	Content Delivery Network — serves content from nearby servers
Latency	Delay in data travel (lower = faster)
Fault Tolerance	System keeps running even when parts fail
High Availability (HA)	System has minimal downtime, always accessible
Failover	Automatically switching to backup when primary fails
Replication	Copying data to multiple locations for safety
OAC	Origin Access Control — restricts S3 access to CloudFront only
TTL	Time To Live — how long cached content stays before refresh
WAF	Web Application Firewall — filters malicious traffic
DDoS	Distributed Denial of Service attack — floods server with traffic
GDPR	EU law requiring data to stay within specific geographic boundaries
Cache Hit	Content served from edge (fast) — no origin server call
Cache Miss	Content not at edge yet — must fetch from origin (slower)
Data Residency	Legal requirement that data must stay within a country/region
CapEx	Capital Expenditure — upfront hardware costs (on-prem)
OpEx	Operational Expenditure — ongoing usage costs (cloud)

🌍 SECTION 1 — AWS Regions

What is a Region?

An AWS Region is a physical geographic location in the world where AWS has built clusters of data centers. Think of it as a "city" where AWS has its infrastructure.

Key facts:

33+ Regions launched globally
100+ Availability Zones worldwide
400+ Edge Locations and Points of Presence
Serves 245+ countries and territories

Important Region Codes to Know

Region Code	Location
`us-east-1`	N. Virginia (USA) — Default, most services launch here first
`us-west-2`	Oregon (USA)
`eu-west-1`	Ireland
`ap-south-1`	Mumbai 🇮🇳 — Use this for India
`ap-southeast-1`	Singapore
`me-south-1`	Bahrain

🇮🇳 For Indian DevOps work: Always use ap-south-1 (Mumbai) — lowest latency for users in India + data stays in India for compliance.

Region Isolation — How It Works

Region A (Mumbai)          Region B (Singapore)
┌──────────────────┐       ┌──────────────────┐
│   Your App Data  │       │   (Empty)         │
│   AZ-a, AZ-b    │  ✗ NOT automatically │  AZ-a, AZ-b  │
│                  │ ─────replicated──▶ │                  │
└──────────────────┘       └──────────────────┘

Critical rules:

A failure in one Region does NOT impact other Regions
Data stays in a Region unless you explicitly move it
YOU are responsible for cross-Region replication if needed

Default vs Opt-In Regions

Type	Which Regions	How to Use
Default-enabled	All Regions before March 20, 2019	Ready to use immediately
Opt-in required	Hong Kong, Bahrain, newer Regions	Must manually enable in Console

Special Restricted Regions

China Regions (Beijing cn-north-1, Ningxia cn-northwest-1)

Require a separate AWS China account entirely
Standard AWS accounts cannot access these

GovCloud (US)

Isolated Regions for US government agencies
For sensitive workloads under FedRAMP, ITAR, CJIS regulations
Not accessible to regular accounts

💡 Interview Tip: If asked "Can I access the China region with my regular AWS account?" — Answer is NO. You need a separate AWS China account operated by local partners.

🏢 SECTION 2 — Availability Zones (AZs)

What is an AZ?

An Availability Zone is one or more discrete data centers within a Region — each with its own independent:

Power supply
Cooling systems
Physical security
Networking

Think of AZs as separate buildings in the same city — close enough to communicate instantly, but isolated enough that a fire in one doesn't affect another.

AZ Structure Inside a Region

Region: ap-south-1 (Mumbai)
│
├── AZ: ap-south-1a
│     ├── Data Center 1 (thousands of servers)
│     ├── Data Center 2
│     └── Data Center 3
│
├── AZ: ap-south-1b
│     ├── Data Center 1
│     ├── Data Center 2
│     └── Data Center 3
│
└── AZ: ap-south-1c
      ├── Data Center 1
      ├── Data Center 2
      └── Data Center 3

All AZs ←──── High-bandwidth, low-latency fiber-optic links ────→ All AZs

Key AZ Facts

Each AZ is physically separated by many kilometers (but within 100 km of each other in the Region)
Connected via high-speed fiber-optic links for near-synchronous replication
Designed to survive: lightning, tornadoes, earthquakes, flooding
AWS recommends: always deploy across at least 2 AZs

Why Does This Matter for DevOps?

❌ Single AZ Deployment:
   App Server → AZ-a only
   If AZ-a fails → App goes down → Users affected ❌

✅ Multi-AZ Deployment:
   App Server → AZ-a + AZ-b (Load Balanced)
   If AZ-a fails → Traffic automatically routes to AZ-b → App stays up ✅

This is High Availability (HA) — one of the most important concepts in Cloud/DevOps/SRE roles.

🏭 SECTION 3 — Data Centers (The Physical Foundation)

Data centers are the actual physical buildings containing the servers, storage, and networking equipment that power everything you do on AWS.

AWS Data Center Design Principles

Principle	What It Means
Site Selection	Locations chosen to minimize flood, earthquake, lightning risk
Redundant Design	Systems anticipate failure and keep running
Critical Backups	Key components backed up across multiple AZs
Capacity Monitoring	Continuous monitoring to deploy infrastructure proactively
Location Secrecy	Exact addresses never disclosed publicly
Automated Failover	Traffic moves away from failed areas instantly — no human needed
Custom Equipment	AWS uses custom-built network hardware from multiple manufacturers

💡 AWS never publicly discloses the exact location of its data centers — this is a security measure.

📡 SECTION 4 — Edge Locations & Points of Presence (PoPs)

The Problem Without Edge Locations

Imagine your server is in Mumbai (ap-south-1) but a user in London wants to download a 50MB video from your website.

User (London) ──── long internet journey ──── Server (Mumbai)
                      ~200-300ms latency

Every time they want that video, it travels halfway around the world. Slow and expensive.

The Solution — Edge Locations / PoPs

AWS has 400+ Edge Locations scattered across major cities worldwide. These act as cache servers — storing copies of popular content close to users.

User (London) ──── Edge Location (London) ──── Server (Mumbai)
                      ~10-30ms latency           (Only first request)

After the first request, the video is cached at the London edge location. Every subsequent user in London gets it from the edge — fast!

Edge Locations vs Regions vs AZs

Infrastructure	Purpose	Examples
Region	Full compute, storage, all services	Mumbai, N. Virginia
AZ	Isolated data centers within Region	ap-south-1a, ap-south-1b
Edge Location / PoP	CDN caching only, no compute	400+ cities worldwide

Regional Edge Caches

A middle layer between origin and edge locations:

Origin Server (Mumbai) 
      ↕ (infrequent)
Regional Edge Cache (e.g., Singapore)
      ↕ (occasional)
Edge Location (Mumbai city, Bangalore, Chennai)
      ↕ (constant, fast)
End Users

Larger cache than individual edge locations
Stores content that's popular regionally but not hot enough for every edge location
Reduces load on origin servers significantly

Services That Use Edge Locations

Amazon CloudFront — CDN (content delivery)
Amazon Route 53 — DNS resolution
AWS Shield — DDoS protection
AWS WAF — Web application firewall

⚡ SECTION 5 — Latency & Fault Tolerance

Latency

Definition: The time (in milliseconds) for data to travel from source to destination.

Low Latency  = Fast response = Happy users
High Latency = Slow response = Users leave your site

How AWS reduces latency:

Choose the Region closest to your users
Use Edge Locations for static content (CloudFront)
Use CloudPing (cloudping.info) to measure real latency to each Region before deciding

Latency comparison (rough):

Mumbai → Mumbai Edge Location:    ~5-20ms   (CDN cache hit)
Mumbai → Mumbai Region:           ~20-50ms  (same country)
Mumbai → Singapore Region:        ~60-100ms (nearby)
Mumbai → N. Virginia Region:      ~180-220ms (far)

Fault Tolerance

Definition: The ability of a system to keep running correctly even when one or more components fail.

Think of a car with a spare tyre — if one tyre punctures, you can still drive. Fault tolerance is the spare tyre for your infrastructure.

How AWS Achieves Fault Tolerance

Multi-AZ Deployment:

Deploy across 2+ AZs in one Region
If AZ-a fails → AZ-b keeps serving traffic
Synchronous replication via dedicated fiber links

Multi-Region Deployment:

Run services across 2+ Regions
Protects against entire regional disasters
Also reduces global latency by serving from nearest Region

Elastic Load Balancing (ELB):

Distributes incoming traffic across healthy instances
Instantly reroutes away from failed nodes/AZs
No manual intervention needed

Automated Failover:

AWS detects failure automatically
Traffic moves to healthy infrastructure instantly
Minimal or zero downtime

🗺️ SECTION 6 — Region Selection Strategy (4 Factors)

Choosing the wrong Region can cause compliance violations, high costs, and slow performance. Always evaluate all 4 factors:

Factor 1 — Data Governance & Legal Requirements (MOST IMPORTANT)

Check this FIRST before anything else.

Some countries/industries legally require data to stay within borders
EU's GDPR restricts where EU citizen data can be stored
Indian financial data regulations may require data in India

Example: A company serving EU customers must store their data in an EU Region (like Frankfurt eu-central-1 or Ireland eu-west-1) — NOT in US East.

Action: Always validate regulatory requirements with your legal/compliance team before choosing a Region.

Factor 2 — Proximity to End Users (Latency)

Deploy in the Region closest to your users
Use cloudping.info to measure actual latency from your location to each Region

Example: Your app has 90% Indian users → ap-south-1 (Mumbai) is the right choice.

Factor 3 — Service & Feature Availability

Not every AWS service exists in every Region
New services typically launch in us-east-1 first, then expand
Always verify the services you need are available in your chosen Region

Example: Bedrock (AWS's AI service) launched in limited Regions first. If you needed it, you had to use those specific Regions.

Check: AWS Regional Services List

Factor 4 — Pricing Variation

The same service costs differently per Region. AWS pricing example:

Region	t3.medium Linux On-Demand
US East (Ohio)	$0.0416/hr
Asia Pacific (Tokyo)	$0.0544/hr
Asia Pacific (Mumbai)	varies

Also consider: Data transfer costs between Regions can add up significantly for high-traffic applications.

Region Selection Decision Flow

1. Is there a legal/compliance requirement for data location?
   YES → That Region is mandatory, no choice
   NO  → Continue ↓

2. Where are most of your users?
   → Choose nearest Region for low latency

3. Do your required AWS services exist in that Region?
   YES → Continue ↓
   NO  → Pick alternative Region where services exist

4. Compare pricing between shortlisted Regions
   → Pick the most cost-effective option

✅ SECTION 7 — Benefits of AWS Global Infrastructure

The 3 pillars that make AWS infrastructure fundamentally different from on-premise:

1. Elastic and Scalable

Resources adjust dynamically to demand
Scale up for Black Friday, scale back down in January
No need to buy hardware for peak capacity

2. Fault Tolerant

Built-in redundancy across AZs and Regions
Critical components backed up automatically
Traffic reroutes away from failures without human intervention

3. Highly Available

Designed for continuous uptime
Minimal to zero human intervention for maintenance
Multiple AZs provide redundant paths and instant failover

Traditional On-Premise:
  Single server → Server fails → Everything down → Engineers scramble → Hours of downtime

AWS Multi-AZ:
  AZ-a fails → AZ-b auto-serves → Users notice nothing → 0 downtime

🌐 SECTION 8 — CloudFront & CDN (Practical Demo)

What is a CDN?

CDN = Content Delivery Network

A global network of servers that store cached copies of your content and serve them from locations close to your users.

Without CDN: Every user fetches from your origin server (slow if far away)

With CDN: Users fetch from the nearest cache server (fast, always)

What is AWS CloudFront?

CloudFront is AWS's CDN service. It sits in front of your origin (S3 bucket, EC2 server, etc.) and:

Caches your content at 400+ Edge Locations worldwide
Serves users from the nearest edge location
Enforces HTTPS
Provides DDoS protection (AWS Shield integration)
Filters malicious traffic (AWS WAF integration)

Request Flow Comparison

Without CloudFront:

User (Mumbai) ──── Internet ──── Origin Server (US East)
                                    ↑
                          500ms - 2000ms latency
                          Every request hits this server
                          No caching, no protection

With CloudFront:

User (Mumbai) ──── Edge Location (Mumbai PoP) ──── Origin (US East)
                         ↑                              ↑
                   CACHE HIT: 20-100ms           Only first request
                   Subsequent: served from edge   Cache TTL controls refresh

Cache Hit vs Cache Miss

First Request (Cache MISS):
  User → Edge → "Not cached here" → Origin → Content fetched → Stored at Edge → Sent to User

All Subsequent Requests (Cache HIT):
  User → Edge → "Found in cache!" → Instantly served to User
  (Origin server never contacted — saves cost and reduces load)

Response headers tell you which happened:

Header	Value	Meaning
`x-cache`	`Hit from cloudfront`	Served from edge cache ✅
`x-cache`	`Miss from cloudfront`	Fetched from origin ⏳
`x-amz-cf-pop`	`BOM78-P1`	Mumbai edge location served you
`age`	`3421`	Content cached for 3421 seconds

CloudFront Lab Setup (Step-by-Step)

Step 1: Create S3 Bucket as Origin

AWS Console → S3 → Create Bucket
Name: cloudfront-demo-lab-2026
Region: us-east-1 (N. Virginia)
Uncheck "Block all public access" → Acknowledge
Upload: index.html (or any image)
Set object permissions: Make Public via ACL
Properties tab → Enable Static Website Hosting

Step 2: Test Direct S3 Access (Baseline)

URL: http://bucket.s3-website-us-east-1.amazonaws.com/index.html
Open DevTools → Network tab → Reload
Note: Response time ~500ms-2000ms (from India)
Note: No x-cache header (no CloudFront)
Note: HTTP only, no HTTPS

Step 3: Create CloudFront Distribution

AWS Console → CloudFront → Create Distribution

Setting                  Value
─────────────────────────────────────────────
Origin Domain            [your S3 bucket]
Origin Access            OAC (Origin Access Control)
Viewer Protocol          Redirect HTTP → HTTPS
Allowed Methods          GET, HEAD
Cache Policy             CachingOptimized (AWS Managed)
Price Class              All Edge Locations
Default Root Object      index.html
WAF                      Enable (optional)

⏱ Wait 5-15 minutes for global deployment

Step 4: Test CloudFront Access

URL: https://d1xxxx.cloudfront.net/index.html
First request → x-cache: Miss from cloudfront (fetches from origin)
Reload → x-cache: Hit from cloudfront (served from edge!)
Response time → 20-100ms (vs 500-2000ms without CDN)

Step 5: Restrict Direct S3 Access (OAC)

OAC = Origin Access Control
Forces all requests through CloudFront only
Direct S3 URL becomes blocked → users must use CloudFront URL
This adds security + ensures all requests get CDN benefits

Normal S3 vs CloudFront Comparison

Factor	Normal S3 Access	CloudFront
Latency	500ms–2000ms+	20ms–100ms
Caching	None	Edge cache (TTL configurable)
HTTPS	Not default	Enforced by default
Origin Load	Every request hits origin	Cache hits skip origin
DDoS Protection	None	AWS Shield + WAF
Geographic Opt.	Fixed region	Nearest edge served
Cost (repeat req.)	Charged per request	Lower with edge caching
Security	Bucket policy only	OAC + WAF + Signed URLs

💡 DevOps Use Case: When deploying a static website, React app, or serving images/videos at scale — always put CloudFront in front of your S3. It's almost always cheaper and significantly faster for global users.

👤 SECTION 9 — IAM Practical Recap

What Was Done in This Session

During this session, the IAM setup from Class 1 was completed practically. Here's the complete picture of what a secure IAM setup looks like:

The IAM Hierarchy

AWS Root Account (your email)
│   ├── ⚠️ LOCK THIS DOWN — MFA enabled, used ONLY for billing/account emergencies
│
├── IAM Groups
│     ├── AdminGroup → AdministratorAccess policy
│     ├── DevOpsGroup → PowerUserAccess policy
│     └── ReadOnlyGroup → ReadOnlyAccess policy
│
├── IAM Users
│     ├── tejas-admin → member of AdminGroup
│     ├── dev-user → member of DevOpsGroup
│     └── auditor → member of ReadOnlyGroup
│
└── IAM Roles (for services, not humans)
      ├── EC2-S3-Role → EC2 can access S3
      └── Lambda-DynamoDB-Role → Lambda can write to DynamoDB

IAM Policy Types

AdministratorAccess — Full control over everything

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

Use for: Learning, your personal admin user

PowerUserAccess — Everything except IAM management

Use for: Developers who deploy but shouldn't manage access

ReadOnlyAccess — View everything, change nothing

Use for: Auditors, monitoring dashboards

Principle of Least Privilege (Always Apply This)

Give users the minimum permissions needed to do their job — nothing more.

❌ Wrong: Give all developers AdministratorAccess "just in case"

✅ Right: 
   Developer needs to deploy EC2 + S3? → Give only EC2 + S3 permissions
   Developer needs to read logs? → Give CloudWatch ReadOnly only
   Developer needs billing info? → Give Billing ReadOnly only

IAM Best Practices Checklist

✅ Root account MFA enabled
✅ IAM user created for all daily tasks (never use Root)
✅ MFA enabled on IAM user too
✅ Strong password policy set
✅ Access keys rotated regularly (if used)
✅ Never commit access keys to GitHub
✅ Use IAM Roles for EC2/Lambda/services (not access keys)
✅ Audit IAM users periodically — remove unused accounts
✅ Use Groups to manage permissions (not individual user policies)
✅ Download and securely store credentials CSV

💰 SECTION 10 — Cost Prevention Checklist (IMPORTANT!)

Accidental AWS charges are the #1 beginner mistake. Here's how to stay safe:

Set Up Billing Alerts (Do This Now!)

AWS Console → Billing → Budgets → Create Budget
→ Set monthly limit (e.g., $5 or $10)
→ Alert email when 80% of budget reached
→ Alert email when 100% reached

Free Tier Monitoring

AWS Console → Billing → Free Tier
→ See what you've used vs what's free
→ Check regularly — especially EC2 hours and S3 storage

After Every Lab Session — Cleanup Checklist

EC2 Instances:
  ✅ STOP instances when not learning (saves compute cost)
  ✅ TERMINATE when done with lab permanently

S3 Buckets:
  ✅ Delete test files and buckets after labs
  ✅ S3 costs per GB stored — don't leave large files

CloudFront:
  ✅ Disable distribution when not in use
  ✅ Delete after lab complete

RDS / Databases:
  ✅ STOP or DELETE RDS instances after labs
  ✅ RDS costs even when idle

IAM Access Keys:
  ✅ Delete unused access keys
  ✅ Never leave programmatic keys lying around

Billing Check (Weekly Habit):
  ✅ AWS Console → Billing → Check current month's charges
  ✅ Any unexpected charge → investigate immediately

Free Tier Limits (12 months)

Service	Free Tier Limit	Watch Out For
EC2	750 hrs/month (t2.micro)	Running 2+ instances = double hours
S3	5 GB storage + 20K GET requests	Large uploads, frequent requests
Lambda	1M requests/month	Usually safe for learning
RDS	750 hrs/month (db.t2.micro)	Easy to forget and leave running
CloudFront	1 TB data transfer/month	Usually safe for labs
Data Transfer OUT	100 GB/month	Serving large files to internet

🧠 INTERVIEW QUESTIONS

Easy

Q1: What is the difference between a Region, AZ, and Edge Location?

A: A Region is a geographic area (e.g., Mumbai). An AZ is one or more isolated data centers within a Region with independent power and networking. An Edge Location is a cache server in a major city used by CloudFront/Route 53 for low-latency content delivery — not for compute.

Q2: How many AZs does a Region typically have?

A: A minimum of 2, typically 3. For example, Mumbai (ap-south-1) has 3 AZs: a, b, c.

Q3: What is CloudFront and what problem does it solve?

A: CloudFront is AWS's CDN (Content Delivery Network). It solves the problem of high latency for users far from your origin server by caching content at 400+ edge locations worldwide, serving users from the nearest one.

Q4: What is the difference between a Cache Hit and Cache Miss in CloudFront?

A: A Cache Hit means the content was found at the edge location and served instantly without contacting the origin. A Cache Miss means the edge location didn't have the content yet and had to fetch it from the origin server.

Q5: What are the 4 factors for choosing an AWS Region?

A: 1) Data Governance/Legal compliance, 2) Proximity to end users (latency), 3) Service/feature availability in that Region, 4) Pricing variation between Regions.

Intermediate

Q6: What is Fault Tolerance and how does AWS achieve it?

A: Fault Tolerance is the ability to keep operating when components fail. AWS achieves it through Multi-AZ deployments (data replicated across AZs), Elastic Load Balancing (routes traffic away from failed nodes), and Automated Failover (systems detect failure and reroute traffic automatically).

Q7: Why is us-east-1 special compared to other Regions?

A: us-east-1 (N. Virginia) is AWS's oldest and largest Region. New services and features always launch there first before expanding to other Regions. It also has the broadest service availability and some of the lowest prices.

Q8: What is OAC in CloudFront and why is it used?

A: OAC (Origin Access Control) restricts access to your S3 bucket so that only CloudFront can access it — users cannot bypass CloudFront and access the S3 URL directly. This enforces security, ensures all traffic benefits from CDN caching, and prevents direct origin exposure.

DevOps-Focused

Q9: A company's web app has users in India, Europe, and USA. The origin server is in Mumbai. How would you design the architecture for optimal performance?

A: Use CloudFront with the Mumbai S3/EC2 origin. CloudFront's 400+ edge locations would automatically serve European and US users from their nearest edge, dramatically reducing latency. Static assets (images, CSS, JS) would be cached at edges. For dynamic content, consider Multi-Region Active-Active deployment if budget allows. Also enable AWS WAF and Shield on CloudFront for security.

Q10: What is the difference between High Availability and Fault Tolerance?

A: Fault Tolerance means the system continues operating with zero interruption when components fail (like a plane with multiple engines). High Availability means the system has minimal downtime and can recover quickly from failure (like RAID storage — brief interruption but quick recovery). In practice, AWS Multi-AZ gives High Availability, while Multi-Region active-active gives Fault Tolerance.

⚡ QUICK REVISION CARD

AWS GLOBAL NUMBERS
├── 33+ Regions globally
├── 100+ Availability Zones
├── 400+ Edge Locations / PoPs
└── 245+ countries served

REGION SELECTION (4 Factors)
├── 1. Legal/Compliance (check FIRST)
├── 2. Proximity to users (latency)
├── 3. Service availability in Region
└── 4. Pricing (varies by Region)

AZ KEY FACTS
├── Min 2 per Region (usually 3)
├── Independent power + networking
├── Within 100km of each other
└── Connected via low-latency fiber

CLOUDFRONT
├── CDN = cache content near users
├── 400+ edge locations
├── Cache Hit → served from edge (fast, ~20ms)
├── Cache Miss → fetched from origin (slow, first time only)
└── OAC = blocks direct S3 access, forces CloudFront

IAM BEST PRACTICES
├── Root → Lock away, MFA, never for daily use
├── IAM User → All daily tasks
├── Least Privilege → Minimum required permissions
└── Groups → Manage permissions at group level

COST PREVENTION
├── Set billing alerts (do this NOW)
├── Stop EC2 when not in use
├── Delete CloudFront/S3 after labs
├── Check Free Tier usage regularly
└── Never leave RDS running idle

📚 KEY TERMS GLOSSARY (Session 2)

Term	Definition
Region	Geographic AWS infrastructure cluster (e.g., Mumbai)
AZ	Isolated data center(s) within a Region with independent power/network
Edge Location	CDN cache server in major cities, used by CloudFront/Route53
Regional Edge Cache	Mid-tier cache between origin and edge, for less popular content
PoP	Point of Presence — AWS nodes in major cities for low-latency delivery
CloudFront	AWS's Content Delivery Network service
CDN	Content Delivery Network — global cache network
Cache Hit	Content found at edge — served instantly, no origin call
Cache Miss	Content not at edge yet — must fetch from origin
TTL	Time To Live — how long content stays cached before refresh
OAC	Origin Access Control — only CloudFront can access S3
Latency	Time for data to travel from source to destination
Fault Tolerance	System continues operating despite component failures
High Availability	System maintains uptime with minimal downtime
Failover	Automatic switch to backup when primary fails
GDPR	EU data privacy law requiring data to stay within EU
GovCloud	Isolated AWS Regions for US government workloads
Data Residency	Legal requirement for data to stay in a specific country
WAF	Web Application Firewall — filters malicious web traffic
DDoS	Distributed Denial of Service — flood attack on servers

🔬 PRACTICE QUESTIONS

A startup in Bangalore is building an app for Indian users. Which Region should they choose and why?
Your app is deployed in only one AZ. What is the risk, and how would you fix it?
You access a CloudFront URL and see x-cache: Miss from cloudfront. What does this mean? What will you see on the second request?
A client says "our hospital patient data cannot leave India." What AWS concept does this relate to, and which Region would you recommend?
What is the difference between an Edge Location and an Availability Zone?
You check your AWS bill and see unexpected S3 charges. List 3 things you would check first.
Why does AWS recommend enabling MFA on both Root AND IAM users?

AWS Regions — More Than Just Geography

An AWS Region isn't just a dot on a map. It's a fully isolated infrastructure cluster — a failure in Mumbai has zero impact on Singapore. And critically: data doesn't move between Regions unless you make it happen.

For India specifically, ap-south-1 (Mumbai) is the obvious choice — lowest latency for Indian users, and data stays within India for compliance.

How to choose a Region? 4 factors:

Legal/compliance requirements (check this FIRST — GDPR, RBI, etc.)
Proximity to your users (use cloudping.info to measure actual latency)
Service availability (not all services exist in every Region)
Pricing (same service, different price per Region)

AZs — Your App's Safety Net

Within each Region, AWS has multiple Availability Zones — isolated data centers with independent power and networking. They're close enough to communicate instantly but isolated enough that one failing doesn't affect the others.

Deploying across 2 AZs = High Availability. If AZ-a goes down, AZ-b serves your users with zero interruption. This is one of the most fundamental DevOps architecture decisions.

CloudFront — CDN Finally Makes Sense

Here's what I understood from the demo:

Without CloudFront: Every user, regardless of location, hits your origin server. A user in London accessing your Mumbai server waits 200-300ms. Every. Single. Time.

With CloudFront: Their first request goes to the nearest edge location (say, London). CloudFront fetches from your Mumbai origin and caches it at the London edge. Every user after that in London gets it in 20-30ms.

The response header x-cache: Hit from cloudfront tells you you're getting the cached version. First request = Miss, every request after = Hit. That's CDN in action.

Key Takeaways

AWS has 33+ Regions, 100+ AZs, and 400+ Edge Locations
Always evaluate 4 factors before choosing a Region: legal, latency, service availability, pricing
Multi-AZ = High Availability. Never deploy production in a single AZ
CloudFront = CDN. It caches content near users, reduces origin load, enforces HTTPS, and adds DDoS protection
OAC ensures users can only access your content through CloudFront — not directly from S3
Set billing alerts before anything else — accidental charges are a real beginner trap