DEV Community

Cover image for AWS S3 Deep Dive — Objects, Encryption, Bucket Policies & Everything In Between
Tejas Shinkar
Tejas Shinkar

Posted on

AWS S3 Deep Dive — Objects, Encryption, Bucket Policies & Everything In Between

#beginners #devops #cloud #aws

Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session goes deep into S3 — the most widely used AWS service and a guaranteed topic in every AWS certification and DevOps interview.

📋 Topics Covered

# Topic Type
1 Why S3 Matters + Quick Recap Concept
2 JSON & YAML — Why They Matter in AWS Concept + Interview
3 API Calls in AWS (GET, PUT, DELETE) Concept + DevOps
4 Region Selection — Why It Matters for S3 Concept + Cert
5 S3 Buckets — Rules & Constraints Concept + Lab
6 S3 Objects — What They Actually Are Concept + Cert
7 S3 Object URL Structure Concept + Lab
8 S3 Storage Classes Concept + Cert
9 S3 Versioning Concept + Lab
10 S3 Encryption — SSE-S3, SSE-KMS, SSE-C, CSE Concept + Cert
11 S3 Bucket Keys — KMS Cost Optimization Concept + Cert
12 Bucket Policies — Ownership & ACL Settings Concept + Interview
13 S3 Access Control — Policies, ACLs, Public Access Concept + Cert
14 Explicit Deny — The Override Rule Concept + Interview
15 AWS CloudTrail + S3 Audit Logging Concept + DevOps
16 VPC — Brief Introduction Concept
17 S3 CLI Commands Practical
18 boto3 — S3 with Python Practical
19 Interview Questions Interview
20 Practice Tasks Practice

Why S3 Matters

S3 (Simple Storage Service) is arguably the most important AWS service to understand — it appears in almost every architecture, every certification, and every DevOps workflow. Backups, static websites, data lakes, CI/CD artifacts, ML training data, log archives — S3 underpins all of it.

Before diving into encryption and policies, two foundational things from this session need to be covered first: JSON/YAML code and API calls — because everything in S3 (and AWS) is built on top of these.

JSON & YAML — Why Interviewers Ask You to Write Them

In real organizations, nobody clicks through the AWS Console to create resources. Everything is written as code — JSON or YAML — and executed via CLI, CloudFormation, or Terraform. Interviewers increasingly ask candidates to write infrastructure code on the spot (sometimes called "vibe coding").

JSON (JavaScript Object Notation):

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "MyEC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": "ami-0abcdef1234567890",
        "InstanceType": "t2.micro",
        "KeyName": "my-key-pair",
        "Tags": [
          { "Key": "Name", "Value": "MyServer" }
        ]
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

YAML (Yet Another Markup Language) — same thing, cleaner to read:

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-0abcdef1234567890
      InstanceType: t2.micro
      KeyName: my-key-pair
      Tags:
        - Key: Name
          Value: MyServer
Enter fullscreen mode Exit fullscreen mode

Key differences:

JSON YAML
Syntax Curly braces {}, quotes "", commas Indentation, colons :, cleaner
Readability Verbose, harder to read Clean, easier to write by hand
Comments ❌ Not supported ✅ Supported with #
Used in AWS for API responses, Lambda, IAM policies CloudFormation, SAM, CodePipeline
Error-prone Missing comma or brace Wrong indentation

S3 Bucket Policy in JSON (what you'll write in real interviews):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPublicRead",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket-name/*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

IAM Policy in YAML (CloudFormation):

Type: AWS::IAM::Policy
Properties:
  PolicyName: S3ReadPolicy
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - s3:GetObject
          - s3:ListBucket
        Resource:
          - arn:aws:s3:::my-bucket
          - arn:aws:s3:::my-bucket/*
Enter fullscreen mode Exit fullscreen mode

🎯 Interview tip: Practice writing a basic S3 bucket policy and EC2 IAM role in both JSON and YAML without looking things up. Interviewers genuinely ask for this now.

API Calls in AWS — How Everything Actually Works

Every click in the AWS Console, every CLI command, every boto3 call — all of it is ultimately an HTTP API call under the hood. Understanding this is essential for DevOps and automation work.

AWS uses standard HTTP methods:

Method Purpose S3 Example
GET Read / retrieve data Download an object, list buckets
PUT Create or replace Upload an object, create a bucket
POST Create or trigger Initiate multipart upload
DELETE Remove Delete an object or bucket
HEAD Get metadata only Check if object exists without downloading

EC2 → S3 interaction via API:

EC2 Instance
    │
    │  PUT /my-bucket/backup.zip  (upload)
    │  GET /my-bucket/config.json (download)
    │  DELETE /my-bucket/old.log  (delete)
    ▼
S3 API Endpoint: https://s3.ap-south-1.amazonaws.com
    │
    ▼
S3 Bucket (my-bucket)
Enter fullscreen mode Exit fullscreen mode

In practice, your application on EC2 uses the AWS SDK (boto3 in Python, or the Java/Node SDK) which converts your code calls into these signed HTTP API requests automatically.

AWS Signature Version 4 — all API calls must be signed with your credentials so AWS can verify identity. The SDK handles this for you when using IAM roles (recommended) or access keys (not recommended).

🎯 DevOps relevance: When you write a Python script with boto3 to copy files to S3, you're making PUT API calls. When your Lambda function reads from DynamoDB, it's making GET API calls. The console is just a visual wrapper over the same APIs.

Region Selection — Why It Matters for S3

S3 is a global service in name, but buckets are created in a specific Region. Choosing the right region affects:

Factor Impact on S3
Performance Store data in Region closest to your users/EC2 → lower latency
Cost S3 pricing varies by Region (US East is typically cheapest)
Data Transfer Transferring data between Regions costs money — keep S3 and EC2 in same Region
Compliance / Regulatory GDPR (EU), RBI (India) may legally require data in specific regions
Disaster Recovery Cross-Region Replication (CRR) copies to another Region for DR

💡 Practical rule: Always create your S3 bucket in the same Region as the EC2 instances or services that will use it. Cross-region data transfer is charged per GB and adds latency.

S3 Buckets — Rules & Constraints

A bucket is the container for your objects in S3. Think of it like a top-level folder — except with globally unique naming rules and no nesting of buckets inside buckets.

Bucket Naming Rules 

✅ Must be globally unique — no two buckets in the world can share a name
✅ 3 to 63 characters long
✅ Lowercase letters, numbers, and hyphens only
✅ Must start with a letter or number
✅ Must not start with "xn--" (reserved)
✅ Must not end with "-s3alias"
❌ No uppercase letters
❌ No underscores
❌ No IP address format (e.g., 192.168.1.1)
❌ No spaces
Enter fullscreen mode Exit fullscreen mode

Good names: tejas-dev-backups, company-prod-logs-2026, my-static-site-assets

Bad names: My_Bucket, 123.456.789.0, TejasBucket

Key Bucket Facts

  • One Region per bucket — created in a specific Region, stays there
  • No size limit on total bucket storage — scales infinitely
  • 5 TB max per individual object — single file can't exceed 5TB
  • Soft limit of 100 buckets per account (can request increase to 1000)
  • No nested buckets — you cannot put a bucket inside a bucket
  • Bucket names in URLs — because names are globally unique, they're used directly in URLs

S3 Objects — What They Actually Are

An object is any file stored in S3. Unlike a traditional file system, S3 is flat — there are no real folders (just prefixes that look like folders).

Object Components

Component Description
Key The full "path" of the object. e.g., reports/2026/january/sales.csv
Value The actual file content (bytes)
Version ID Unique ID if versioning is enabled
Metadata Key-value pairs about the object (content-type, custom tags)
ETag MD5 hash of the object (used to verify integrity)
Storage Class Standard, IA, Glacier, etc.

The "Folder" Illusion 

Bucket: my-data
Objects stored as:
  reports/2026/sales.csv    ← key includes the "path"
  reports/2026/revenue.csv
  images/logo.png
  images/banner.jpg
  config.json

There are NO real folders. "reports/2026/" is just a prefix.
The console shows them as folders for readability — but it's flat storage underneath.
Enter fullscreen mode Exit fullscreen mode

Object Size Limits

Upload type Size limit When to use
Single PUT upload Up to 5 GB Small to medium files
Multipart upload Up to 5 TB Files larger than 100 MB (recommended)
Minimum part size 5 MB For multipart upload chunks

✅ Best practice: Use multipart upload for any file over 100 MB. It's faster (parallel chunks), more reliable (resume on failure), and required above 5 GB.

S3 Object URL Structure

Every object in S3 has a predictable, deterministic URL. Understanding the format is important for linking to files, debugging access issues, and building applications.

URL Formats

Path-style URL (older, being deprecated):

https://s3.<region>.amazonaws.com/<bucket-name>/<object-key>
https://s3.ap-south-1.amazonaws.com/tejas-dev/reports/sales.csv
Enter fullscreen mode Exit fullscreen mode

Virtual-hosted-style URL (current standard):

https://<bucket-name>.s3.<region>.amazonaws.com/<object-key>
https://tejas-dev.s3.ap-south-1.amazonaws.com/reports/sales.csv
Enter fullscreen mode Exit fullscreen mode

Breaking it down:

https://tejas-dev.s3.ap-south-1.amazonaws.com/reports/2026/sales.csv
        │          │   │                        │
        bucket     │   region                   object key (full path)
                   S3 service
Enter fullscreen mode Exit fullscreen mode

URL Validity

An S3 object URL is only valid while:

  1. The object actually exists in the bucket
  2. The object is publicly accessible (or you're using a pre-signed URL with a valid expiry)

If you delete the object, the URL returns a 404 — the URL structure stays the same but there's nothing at that address.

Pre-Signed URLs

For private objects, you can generate a pre-signed URL — a temporary link with an expiry time that grants access without making the object public.

import boto3
s3 = boto3.client('s3')

# Generate a URL valid for 1 hour
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-private-bucket', 'Key': 'secret-file.pdf'},
    ExpiresIn=3600  # seconds
)
print(url)
# https://my-private-bucket.s3.ap-south-1.amazonaws.com/secret-file.pdf?X-Amz-Signature=...&X-Amz-Expires=3600
Enter fullscreen mode Exit fullscreen mode

🎯 Common use case: Share a download link for a private file with a customer for 24 hours — pre-signed URL with ExpiresIn=86400. Link expires automatically.

S3 Storage Classes

S3 isn't one-size-fits-all for storage. Different classes trade cost vs access speed vs retrieval time.

Storage Class Use Case Retrieval Cost
S3 Standard Frequently accessed data Milliseconds Highest
S3 Intelligent-Tiering Unknown or changing access patterns Milliseconds Moderate (auto-tiered)
S3 Standard-IA Infrequent access, needs fast retrieval Milliseconds Lower storage, retrieval fee
S3 One Zone-IA Infrequent, reproducible data, one AZ only Milliseconds Lower than Standard-IA
S3 Glacier Instant Archive, accessed once a quarter Milliseconds Very low
S3 Glacier Flexible Archive, minutes-to-hours retrieval ok Minutes to hours Very low
S3 Glacier Deep Archive Long-term archive, accessed once a year Up to 12 hours Cheapest

Simple decision guide:

Serving to users daily?              → Standard
Don't know access pattern?           → Intelligent-Tiering (auto-manages)
Accessed once a month, need fast?    → Standard-IA
Old logs, rarely accessed?           → Glacier Instant Retrieval
Legal archive, won't need for years? → Glacier Deep Archive
Enter fullscreen mode Exit fullscreen mode

Lifecycle Policies (Auto-Tiering)

Just like EFS, you can set rules to automatically move objects between storage classes:

Day 0    → Upload to Standard
Day 30   → Auto-move to Standard-IA
Day 90   → Auto-move to Glacier Instant
Day 365  → Auto-move to Glacier Deep Archive
Day 2555 → Auto-delete (7 years, for compliance)
Enter fullscreen mode Exit fullscreen mode

🎯 Cert tip: "How do you minimize S3 storage costs for logs that need to be kept for 7 years but rarely accessed after 30 days?" → Lifecycle policy transitioning to Glacier.

S3 Versioning

When versioning is enabled, S3 keeps every version of every object — overwrites create a new version instead of replacing, and deletes add a "delete marker" instead of actually deleting.

Upload file.txt (v1)      → Version ID: aaa111
Upload file.txt (v2)      → Version ID: bbb222  ← current version
Upload file.txt (v3)      → Version ID: ccc333  ← current version

GET file.txt              → returns v3 (current)
GET file.txt?versionId=aaa111 → returns v1
DELETE file.txt           → adds delete marker, v3 still exists
Enter fullscreen mode Exit fullscreen mode

Why versioning matters:

  • Accidental overwrite? Restore previous version instantly
  • Accidental delete? Remove the delete marker to recover
  • Required for Cross-Region Replication
  • Required for S3 Object Lock (compliance)

⚠️ Cost warning: Versioning stores every version of every file. An object updated 100 times = 100 stored copies. Enable Lifecycle Policies to expire old versions, otherwise storage costs grow silently.

S3 Encryption — Four Methods

Encryption in S3 has two dimensions: at rest (stored on disk) and in transit (moving over the network). Transit is always HTTPS — the four methods below are all about at rest.

SSE-S3 (Server-Side Encryption with S3-Managed Keys)

  • AWS manages everything — keys are created, rotated, and used by S3 automatically
  • Enabled by default on all new buckets (since Jan 2023)
  • Zero configuration needed
  • AES-256 encryption
  • You have no control over the keys 
Upload file → S3 auto-encrypts with its own key → stored encrypted
Download → S3 auto-decrypts → you get the file
You never see or touch the key
Enter fullscreen mode Exit fullscreen mode

Use when: You need encryption but don't have compliance requirements about key control.

SSE-KMS (Server-Side Encryption with AWS KMS Keys)

Used mostly

Upload flow:

You upload file.pdf
      ↓
S3 sends request to AWS KMS: "Give me a data encryption key"
      ↓
KMS generates a data key (encrypted + plaintext versions)
      ↓
S3 uses plaintext key to encrypt your file
      ↓
S3 stores: encrypted file + encrypted key (plaintext key discarded)
      ↓
Your file sits on disk: fully encrypted ✅
Enter fullscreen mode Exit fullscreen mode

Download flow:

You request file.pdf
      ↓
S3 checks your IAM permissions (must have s3:GetObject)
      ↓
S3 checks your KMS permissions (must have kms:Decrypt)
      ↓
S3 sends encrypted key to KMS: "Decrypt this"
      ↓
KMS decrypts the key → returns plaintext key
      ↓
S3 uses plaintext key to decrypt your file
      ↓
You receive the original file ✅
Enter fullscreen mode Exit fullscreen mode

Why SSE-KMS over SSE-S3?

  • You control the KMS key (create, rotate, disable, delete)
  • Full audit trail — every KMS key usage logged in CloudTrail
  • Granular access control — you can restrict who can use the key via KMS key policy
  • Required by many compliance standards (HIPAA, PCI-DSS, FedRAMP)

Downside: Every object upload/download = one KMS API call. KMS has a cost per API call and rate limits.

SSE-C (Server-Side Encryption with Customer-Provided Keys)

  • You provide your own encryption key with each request
  • AWS uses it to encrypt/decrypt, but never stores your key
  • You are 100% responsible for key management 
Upload: You send (file + your-key) → S3 encrypts → discards your-key
Download: You send (object-key + your-key) → S3 decrypts → returns file
Lose your key = lose your data permanently (no recovery)
Enter fullscreen mode Exit fullscreen mode

Use when: You have strict regulatory requirements that AWS must never hold your encryption key even temporarily.

CSE (Client-Side Encryption)

  • You encrypt the file before uploading it to S3
  • S3 just stores whatever bytes you send — it has no idea the content is encrypted
  • You manage all encryption/decryption on your side 
Your app: file → encrypt locally → send encrypted bytes → S3 stores them
Your app: download encrypted bytes → decrypt locally → original file
AWS sees only ciphertext — never the original data
Enter fullscreen mode Exit fullscreen mode

Use when: Maximum data confidentiality — even AWS employees cannot read your data.

Encryption Comparison

Method Who manages key AWS sees plaintext? Audit trail Use when
SSE-S3 AWS (automatic) Internally yes Basic Default, no compliance needs
SSE-KMS You (via KMS) Internally yes Full (CloudTrail) Compliance, audit requirements
SSE-C You Never None Strict key control
CSE You Never None Maximum confidentiality

S3 Bucket Keys — Reducing KMS Costs by 99%

With SSE-KMS, every single object upload/download triggers a KMS API call. For buckets with millions of objects, this generates millions of KMS calls — which have both a per-call cost and API rate limits.

S3 Bucket Keys solve this:

Without Bucket Key:

100 object uploads → 100 KMS API calls → $$$
Enter fullscreen mode Exit fullscreen mode

With Bucket Key enabled:

S3 generates one bucket-level data key from KMS
That key is cached in S3 for a time period
100 object uploads → 1 KMS API call + 99 local operations → $ (99% fewer calls)
Enter fullscreen mode Exit fullscreen mode

Analogy: Instead of going to the bank (KMS) to get cash for every purchase, you withdraw a stack of cash once and use it locally for many purchases.

  • Reduces KMS API costs by up to 99%
  • Reduces KMS request rate (avoids throttling)
  • Transparent — encryption strength is unchanged
  • Enable it when creating the bucket or in bucket properties

🎯 Cert tip: "A company uses SSE-KMS on an S3 bucket with millions of daily uploads and is hitting KMS rate limits. What's the fix?" → Enable S3 Bucket Keys.

AWS CloudTrail + S3 Audit Logging

CloudTrail records every API call made to AWS services — who did what, when, from where. For S3 with KMS encryption, it becomes your full audit trail.

What CloudTrail logs for S3 + KMS:

Who:    IAM user/role that accessed the object
What:   s3:GetObject, s3:PutObject, kms:Decrypt, kms:GenerateDataKey
When:   Timestamp of the request
Where:  Source IP address
Result: Success or failure (and why if failed)
Enter fullscreen mode Exit fullscreen mode

Two types of S3 events in CloudTrail:

  • Management Events — bucket-level operations (CreateBucket, DeleteBucket, PutBucketPolicy) — logged by default
  • Data Events — object-level operations (GetObject, PutObject, DeleteObject) — must be explicitly enabled (generates high volume)

S3 Server Access Logging (separate from CloudTrail):

  • Detailed request logs written directly to another S3 bucket
  • Every HTTP request to the bucket: requester IP, request type, response code, bytes transferred
  • Good for access pattern analysis and billing investigation

🎯 DevOps/Security use case: Enable CloudTrail data events for your S3 bucket to detect unauthorized access. If someone outside your organization downloads a sensitive file, CloudTrail has the full record — user, IP, timestamp, object key.

Bucket Policies — Object Ownership & ACL Settings

When multiple AWS accounts are involved (or when files are uploaded by external parties), there are three ownership models that control who owns uploaded objects and how ACLs work.

Three Object Ownership Settings

1. Bucket Owner Enforced (Recommended)

  • All objects in the bucket are always owned by the bucket owner
  • ACLs are completely disabled
  • All access must be controlled via bucket policies or IAM policies only
  • Simplest and most secure model 
External account uploads file → bucket owner automatically owns it
No ACL confusion, no cross-account ownership issues
Enter fullscreen mode Exit fullscreen mode

2. Bucket Owner Preferred

  • Bucket owner takes ownership IF the uploader includes bucket-owner-full-control ACL
  • ACLs are still enabled
  • Gives flexibility while nudging toward central ownership 
External account uploads with bucket-owner-full-control ACL
→ bucket owner gets ownership ✅
External account uploads without that ACL
→ external account retains ownership ⚠️
Enter fullscreen mode Exit fullscreen mode

3. Object Writer

  • The account that uploads an object owns it (legacy default behavior)
  • Can cause situations where bucket owner cannot read/delete objects in their own bucket
  • ACLs enabled

✅ Best practice: Always use Bucket Owner Enforced for new buckets. It's simpler, removes ACL complexity, and avoids cross-account ownership confusion that has caused real security incidents.

S3 Access Control — Full Picture

S3 has multiple layers of access control that work together:

Request to access S3 object
        │
        ▼
1. IAM Policy — does the requester's IAM user/role allow this?
        │
        ▼
2. Bucket Policy — does the bucket policy allow this request?
        │
        ▼
3. S3 Block Public Access — is public access blocked at account/bucket level?
        │
        ▼
4. ACL (if enabled) — does the object/bucket ACL permit access?
        │
        ▼
Access Granted or Denied
Enter fullscreen mode Exit fullscreen mode

All applicable policies are evaluated together. An explicit Deny at ANY layer = denied, regardless of allows elsewhere.

Bucket Policy

A JSON document attached to the bucket that defines who can access it and what they can do:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowEC2Access",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789:role/EC2-S3-Role"
      },
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "DenyDeleteForEveryone",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

S3 Block Public Access (4 Settings)

AWS added this specifically to prevent accidental data exposure — one of the most common cloud security incidents.

Setting What it blocks
Block public ACLs Rejects PUT requests with public ACLs
Ignore public ACLs Ignores existing public ACLs
Block public bucket policies Rejects bucket policies that grant public access
Restrict public bucket policies Restricts public access even if bucket policy allows it

✅ Best practice: Enable all 4 Block Public Access settings at the account level. Explicitly disable only for specific buckets that genuinely need public access (e.g., static website hosting).

Explicit Deny — The Override Rule

This is one of the most important concepts in all of AWS security — and it applies everywhere, not just S3.

The rule:

Explicit Deny ALWAYS wins over any Allow, from any source
Enter fullscreen mode Exit fullscreen mode

Policy evaluation order:

1. Is there an Explicit Deny anywhere? → DENIED (done, nothing else checked)
2. Is there an Explicit Allow? → ALLOWED
3. Neither? → DENIED (implicit deny is the default)
Enter fullscreen mode Exit fullscreen mode

Why this matters in practice:

Scenario: User has AdministratorAccess (allows everything)
          SCP says: Deny s3:DeleteBucket

Result: User CANNOT delete S3 buckets — despite having AdministratorAccess
The Deny wins. Always.
Enter fullscreen mode Exit fullscreen mode

Real example in a bucket policy:

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": "arn:aws:s3:::critical-data/*",
  "Condition": {
    "Bool": {
      "aws:SecureTransport": "false"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

This denies all S3 actions if the request isn't using HTTPS — even if the user has full S3 permissions elsewhere. The Deny wins.

🎯 Interview tip: "A user has S3FullAccess but can't delete objects from a specific bucket. Why?" → There's likely an explicit Deny somewhere — in a bucket policy, SCP, or permission boundary. Explicit Deny always overrides explicit Allow.

VPC — Brief Introduction

VPC = Virtual Private Cloud — your own logically isolated private network inside AWS.

Think of it as buying a plot of land inside a large city (AWS). Within your plot, you decide the layout — which areas are public-facing, which are private, what gets connected to the internet.

Why it's relevant to S3 here: By default, S3 is accessed over the public internet (via HTTPS). For security, you can set up a VPC Endpoint for S3 — a private connection from your VPC directly to S3 without traffic leaving the AWS network.

Without VPC Endpoint:
EC2 → public internet → S3 (traffic exits AWS network)

With VPC Endpoint (Gateway type):
EC2 → VPC Endpoint → S3 (traffic stays inside AWS backbone)
     No internet gateway needed
     More secure, no data transfer costs for S3
Enter fullscreen mode Exit fullscreen mode

VPC will be covered in depth in a dedicated session. For now: VPC = your private network, and S3 can be accessed privately through it via Gateway Endpoints.

S3 CLI Commands 

# List all buckets
aws s3 ls

# List objects in a bucket
aws s3 ls s3://my-bucket/
aws s3 ls s3://my-bucket/reports/ --recursive

# Create a bucket
aws s3 mb s3://my-new-bucket-name --region ap-south-1

# Upload a file
aws s3 cp local-file.txt s3://my-bucket/uploads/file.txt

# Upload entire folder
aws s3 cp ./my-folder s3://my-bucket/my-folder/ --recursive

# Download a file
aws s3 cp s3://my-bucket/file.txt ./local-file.txt

# Sync local folder to S3 (only uploads new/changed files)
aws s3 sync ./local-folder s3://my-bucket/folder/

# Delete an object
aws s3 rm s3://my-bucket/old-file.txt

# Delete all objects in a folder
aws s3 rm s3://my-bucket/old-folder/ --recursive

# Delete a bucket (must be empty first)
aws s3 rb s3://my-bucket

# Delete bucket and all contents
aws s3 rb s3://my-bucket --force

# Copy between buckets (cross-region)
aws s3 cp s3://source-bucket/file.txt s3://dest-bucket/file.txt \
  --source-region ap-south-1 --region us-east-1

# Set storage class on upload
aws s3 cp file.txt s3://my-bucket/ --storage-class STANDARD_IA

# Generate pre-signed URL (valid 1 hour)
aws s3 presign s3://my-bucket/private-file.pdf --expires-in 3600

# Put bucket policy
aws s3api put-bucket-policy \
  --bucket my-bucket \
  --policy file://policy.json

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled
Enter fullscreen mode Exit fullscreen mode

boto3 — S3 with Python 

import boto3

s3 = boto3.client('s3', region_name='ap-south-1')
s3_resource = boto3.resource('s3')

# ── List buckets ────────────────────────────────────────────
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(f"Bucket: {bucket['Name']} | Created: {bucket['CreationDate']}")

# ── Create bucket ───────────────────────────────────────────
s3.create_bucket(
    Bucket='my-new-bucket-2026',
    CreateBucketConfiguration={'LocationConstraint': 'ap-south-1'}
)

# ── Upload file ─────────────────────────────────────────────
s3.upload_file('local-file.txt', 'my-bucket', 'uploads/file.txt')

# ── Upload with SSE-KMS encryption ─────────────────────────
s3.upload_file(
    'sensitive.pdf',
    'my-bucket',
    'secure/sensitive.pdf',
    ExtraArgs={
        'ServerSideEncryption': 'aws:kms',
        'SSEKMSKeyId': 'arn:aws:kms:ap-south-1:123456789:key/abc-123'
    }
)

# ── Download file ───────────────────────────────────────────
s3.download_file('my-bucket', 'uploads/file.txt', 'downloaded-file.txt')

# ── List objects in bucket ──────────────────────────────────
response = s3.list_objects_v2(Bucket='my-bucket', Prefix='reports/')
for obj in response.get('Contents', []):
    print(f"Key: {obj['Key']} | Size: {obj['Size']} bytes")

# ── Delete object ───────────────────────────────────────────
s3.delete_object(Bucket='my-bucket', Key='old-file.txt')

# ── Generate pre-signed URL ─────────────────────────────────
url = s3.generate_presigned_url(
    'get_object',
    Params={'Bucket': 'my-bucket', 'Key': 'private-doc.pdf'},
    ExpiresIn=3600
)
print(f"Share this URL (valid 1 hour): {url}")

# ── Apply bucket policy ─────────────────────────────────────
import json
policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": [
            "arn:aws:s3:::my-bucket",
            "arn:aws:s3:::my-bucket/*"
        ],
        "Condition": {"Bool": {"aws:SecureTransport": "false"}}
    }]
}
s3.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(policy))
Enter fullscreen mode Exit fullscreen mode

⚡ Quick Revision 

JSON vs YAML
  JSON = braces/quotes/commas, API responses, IAM policies
  YAML = indentation/colons, CloudFormation, SAM
  Both used for infrastructure-as-code — know both

API CALLS
  GET    = read/retrieve
  PUT    = create/replace
  POST   = create/trigger
  DELETE = remove
  HEAD   = metadata only

S3 OBJECT URL
  https://<bucket>.s3.<region>.amazonaws.com/<key>
  Valid only while object exists and access is permitted
  Pre-signed URL = temporary access for private objects

STORAGE CLASSES (cost vs speed)
  Standard          → frequent access, fastest, most expensive
  Intelligent-Tiering → unknown patterns, auto-optimizes
  Standard-IA       → monthly access, fast retrieval fee
  Glacier Instant   → quarterly access, instant retrieval
  Glacier Deep      → annual access, 12hr retrieval, cheapest

ENCRYPTION
  SSE-S3  → AWS manages key, default, zero config
  SSE-KMS → you control key via KMS, audit trail, compliance
  SSE-C   → you provide key per request, AWS never stores it
  CSE     → you encrypt before upload, AWS sees ciphertext only

S3 BUCKET KEYS
  Reduces KMS API calls by 99%
  One bucket-level key cached, used for multiple objects
  Enable for high-volume SSE-KMS buckets

BUCKET OWNERSHIP
  Owner Enforced  = ACLs off, bucket owner owns all (use this)
  Owner Preferred = owner gets ownership if ACL included
  Object Writer   = uploader owns (legacy, avoid)

EXPLICIT DENY
  Always overrides any Allow, from any source
  Evaluation: Explicit Deny → Explicit Allow → Implicit Deny

CLOUDTRAIL
  Logs every API call: who, what, when, where, result
  Data events (object-level) must be explicitly enabled
  Use for security auditing of S3 + KMS access
Enter fullscreen mode Exit fullscreen mode

💼 Interview Questions

Q1: What is the difference between an S3 bucket and an S3 object?
A bucket is the container — like a top-level folder — created in a specific Region with a globally unique name. An object is the actual file stored inside the bucket, uniquely identified by its key (path). The bucket holds the object; the object is the data.

Q2: What makes an S3 object URL unique and when does it stop working?
The URL is formed from the bucket name, region, and object key — all of which are unique. It stops working when the object is deleted or when access permissions are revoked. For private objects, you need a pre-signed URL with an expiry time to share temporary access.

Q3: What is SSE-KMS and how does it work?
SSE-KMS uses AWS Key Management Service for encryption. On upload, S3 requests a data key from KMS, encrypts the file with the plaintext key, and stores both the encrypted file and the encrypted key (discarding the plaintext key). On download, S3 verifies IAM and KMS permissions, asks KMS to decrypt the encrypted key, then decrypts and returns the file. You control the KMS key and every usage is logged in CloudTrail.

Q4: What are S3 Bucket Keys and why would you enable them?
S3 Bucket Keys are a bucket-level data key cached in S3, used to encrypt multiple objects instead of calling KMS for every single operation. This reduces KMS API calls by up to 99%, significantly reducing cost and avoiding KMS rate limit throttling — essential for high-throughput S3 buckets using SSE-KMS.

Q5: What is the difference between SSE-KMS and SSE-C?
SSE-KMS means AWS generates and manages the data keys inside KMS — you control the master key but AWS handles the actual encryption. SSE-C means you provide your own encryption key with every request — AWS uses it and immediately discards it, never storing it. If you lose your SSE-C key, your data is gone forever.

Q6: An IAM user has S3FullAccess but cannot delete objects from a bucket. Why?
There's an explicit Deny somewhere overriding the Allow. It could be in the bucket policy, an SCP at the Organizations level, or a permission boundary. Explicit Deny always wins over any Allow, from any source.

Q7: What is Bucket Owner Enforced and why is it recommended?
It disables ACLs entirely and ensures the bucket owner always owns all objects regardless of who uploaded them. It's the simplest and most secure model — no ACL confusion, no cross-account ownership issues, and access is controlled purely through bucket policies and IAM.

Q8: Write a bucket policy that denies all HTTP (non-HTTPS) access.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::my-bucket",
      "arn:aws:s3:::my-bucket/*"
    ],
    "Condition": {
      "Bool": {"aws:SecureTransport": "false"}
    }
  }]
}
Enter fullscreen mode Exit fullscreen mode

🔬 Practice Tasks

  1. Create and configure a bucket: Create an S3 bucket with versioning enabled, Block Public Access on, and a lifecycle policy that moves objects to Standard-IA after 30 days and Glacier after 90 days.

  2. Write bucket policies: Write a JSON bucket policy that: (a) allows your IAM user to read all objects, (b) allows an EC2 role to upload objects, and (c) denies all non-HTTPS requests.

  3. SSE-KMS lab: Enable SSE-KMS on a bucket. Upload a file. Download it. Open CloudTrail and find the kms:GenerateDataKey and kms:Decrypt events corresponding to your upload and download.

  4. Pre-signed URL: Using boto3, upload a private file to S3, generate a pre-signed URL valid for 5 minutes, open it in your browser, wait 5 minutes, try again — confirm it expires.

  5. CLI practice: Using AWS CLI, sync a local folder to S3, list the objects, update one file locally, sync again, and confirm only the changed file was re-uploaded (not everything).

  6. Versioning test: Enable versioning on a bucket. Upload a file, modify it, upload again with the same key, upload a third time. List all versions. Retrieve v1. Delete the latest version and confirm v2 becomes current.

  7. Write the JSON and YAML: Without looking at notes, write from scratch: (a) a CloudFormation YAML snippet that creates an S3 bucket with versioning, (b) an S3 bucket policy in JSON that makes all objects in /public/* readable by everyone.

AWS Session 7 — S3 Essentials | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer

Top comments (0)