DEV Community: Tejas Shinkar

AWS Database Mini Projects - RDS, Aurora, DynamoDB & ElastiCache

Tejas Shinkar — Fri, 18 Sep 2026 03:41:29 +0000

Five Production-Style Builds Across RDS, Aurora, DynamoDB & ElastiCache

Part of my AWS learning journey. Instead of practicing RDS, Aurora, DynamoDB, and ElastiCache as isolated labs, I built five small connected projects instead, each one following the same production discipline: a dedicated environment, IAM least privilege, validation, failover or performance testing, and cleanup.

Full repository: github.com/tejasshinkar/Aws-Database-mini-projects

Concepts:
AWS RDS and Elasticache: https://dev.to/tejas_shinkar/aws-rds-elastic-cache-managed-databases-multi-az-failover-in-memory-caching-5dce

AWS Aurora and Dynamodb: https://dev.to/tejas_shinkar/aws-aurora-elasticache-patterns-dynamodb-the-complete-data-layer-1k3e

📋 Projects Covered

#	Project	Main AWS Services	Type
1	RDS + EC2 Application	EC2, RDS, IAM, Security Groups, Python/Flask	Lab
2	RDS High Availability + Read Replica	RDS, Multi-AZ, Read Replica, EC2	Lab
3	Aurora High Availability + Failover	Amazon Aurora, RDS	Lab
4	DynamoDB Design + Performance	DynamoDB, Partition/Sort Keys, GSI, LSI	Lab
5	DynamoDB + ElastiCache for Valkey	DynamoDB, ElastiCache for Valkey, EC2, IAM, Python	Lab

Approach Applied Across All Five

Every project in this set followed the same working process rather than five unrelated one-off exercises:

Understand the AWS service or architecture pattern before building it.
Create and configure the required AWS resources.
Apply IAM and network-security controls appropriate to the workload.
Implement or test the actual solution, not just provision it.
Troubleshoot configuration and permission issues as they came up.
Capture evidence through screenshots and test results.
Document the implementation and the cleanup process.
Remove unused AWS resources to avoid unnecessary charges.

Project 1 — RDS + EC2 Application

A Flask application hosted on EC2, connected to an Amazon RDS database. The focus was on wiring the application and database together the way a real deployment would, using IAM and Security Groups to control access rather than hardcoding database credentials into the application.

Skills exercised: RDS configuration and database connectivity, EC2 administration and application hosting, IAM roles and least-privilege access, Security Groups and network troubleshooting.

Project 2 — RDS High Availability + Read Replica

Took the same RDS + EC2 foundation further to explore how RDS handles availability and read scaling in practice: Multi-AZ for automatic failover, and a Read Replica for offloading read traffic from the primary instance.

Skills exercised: RDS Multi-AZ concepts, Read Replica configuration, database high availability and scalability concepts.

Project 3 — Aurora High Availability + Failover

Moved from standard RDS engines to Amazon Aurora specifically to work with its replication model, cluster endpoints, and failover behavior directly, rather than only reading about how Aurora's architecture is supposed to behave.

Skills exercised: Aurora replication and failover concepts, cluster endpoint behavior, database high availability patterns.

Project 4 — DynamoDB Design + Performance

A shift out of the relational world entirely: designing a DynamoDB table's partition keys, sort keys, Global Secondary Indexes, and Local Secondary Indexes, then testing how those design decisions actually affect access patterns and performance under load.

Skills exercised: DynamoDB data modeling and key design, performance testing and access-pattern evaluation, NoSQL schema design trade-offs.

Project 5 — DynamoDB + ElastiCache for Valkey

Added a caching layer in front of DynamoDB and implemented the lazy-loading (cache-aside) pattern along with cache invalidation, the same approach that shows up constantly in systems that can't afford to hit the database on every single read.

Skills exercised: ElastiCache for Valkey integration, lazy loading and cache-aside architecture, cache invalidation strategy, Python automation with the AWS SDK (Boto3).

Repository Structure

aws-database-mini-projects/
│
├── README.md
│
├── 01-rds-ec2-application/
├── 02-rds-ha-read-replica/
├── 03-aurora-ha-failover/
├── 04-dynamodb-design-performance/
└── 05-dynamodb-elasticache-valkey/

Each project directory contains its own README, implementation details, and test evidence where applicable.

Technologies Used

Cloud: Amazon Web Services
Compute: Amazon EC2
Relational Databases: Amazon RDS, Amazon Aurora
NoSQL Database: Amazon DynamoDB
Caching: ElastiCache for Valkey
Networking: VPC, Security Groups, Subnets
Security: AWS IAM
Automation: Python, Boto3
Version Control: Git and GitHub

Key Takeaways

Connecting an EC2-hosted application to a managed database is straightforward technically, but doing it with least-privilege IAM and scoped Security Groups instead of hardcoded credentials is what makes it production-appropriate.
RDS Multi-AZ and Read Replicas solve two different problems, availability and read scaling, and shouldn't be conflated with each other.
Aurora's replication and failover behavior is different enough from standard RDS that it's worth testing directly rather than assuming the concepts transfer one-to-one.
Good DynamoDB performance starts at the schema design stage. Partition key, sort key, and index choices made early determine what access patterns are actually efficient later.
A cache is only as good as its invalidation strategy. Lazy loading solves the "don't cache what nobody asked for" problem, but it has to be paired with a real invalidation approach or it just serves stale data faster.
The habit that mattered most across all five projects wasn't any single AWS service. It was the repeated discipline of scoping access, validating the result, and cleaning up afterward.

These projects were built for learning, experimentation, and portfolio development. Resource configurations are simplified compared with a real production environment, and all AWS resources were deleted or stopped after testing to avoid unexpected costs.

When You Know Something's Wrong But Can't Say Why Yet - Jesse Pinkman Energy

Tejas Shinkar — Thu, 17 Sep 2026 13:40:04 +0000

You know that feeling in a code review where something's off, but you can't point to a specific line and say "this is broken"?

That's Jesse Pinkman energy. The guy who couldn't explain the chemistry, but somehow always knew when a batch was going to go bad before it actually did.

Except in code review, it's usually the other way around. The senior engineer can't always explain the deep architectural reasoning either. Sometimes it's just pattern recognition built from years of watching similar things break in production. And that feeling gets dismissed constantly, because "I don't like this" doesn't sound like proper feedback in a pull request comment.

Here's the reframe worth sitting with. Vague unease during a code review is data. It's just data you haven't translated into words yet.

Why every engineer has felt this at least once

If you've worked in software for more than a few months, you already know this feeling by a dozen other names. A hunch. A nagging doubt. That little itch at the back of your head that shows up right when you're about to click approve.

It usually shows up around a few very specific situations:

A function that technically works but feels like it's holding the whole system together with tape
An architecture decision that solves today's problem while quietly creating three more for next quarter
A "quick fix" that's clearly going to become permanent the moment it merges
Error handling that only covers the happy path, and everyone in the room quietly knows it

None of these show up as a compile error. None of them fail a test. They just sit there, uncomfortable, until either you say something, or you don't, and the code ships anyway.

The silence trap in software engineering teams

Most engineers who get that "something's wrong here" feeling do one of two things.

They stay quiet, because they can't fully articulate it yet and don't want to sound unsure in front of the team. Or they approve the PR anyway, because pushing back without a clean, specific reason feels unfair to the person who wrote it, especially if that person is more senior or under deadline pressure.

Both paths lead to the exact same place. The code ships. The feeling was right. Three weeks later it's an incident report, a rollback, or a very awkward retro, and everyone's trying to reconstruct what actually went wrong, when someone in that original review already half knew.

"Chemistry is the study of change." And so is a good code review, if anyone actually says what they're noticing before it turns into a postmortem.

This is one of the quieter, less talked about parts of engineering team communication. We optimize for clear, actionable, confident feedback, and that's genuinely good advice for most review comments. But the earliest warning signs almost never arrive pre packaged as clear and actionable. They arrive fuzzy. They arrive as a feeling first, and a reason second, if at all.

What the best reviewers actually do differently

The engineers who catch problems early aren't always the ones with the deepest technical knowledge in the room. Sometimes they're just the ones willing to say "I don't know why yet, but can we double check this" before it ships, instead of after it breaks.

You don't need a fully formed argument to raise a concern in a pull request. A few templates that work surprisingly well:

"This feels fragile, can you walk me through what happens if X fails"
"I don't have a specific objection yet, but this reminds me of something that broke before, can we talk it through"
"Genuinely not sure if this is a problem or just unfamiliar to me, flagging it anyway"

None of these require certainty. All of them get the uncertainty out of your head and into the conversation, which is really the whole point of a code review in the first place.

Half the time, just saying the vague thing out loud is enough to resolve it. Either the author explains their reasoning and the feeling settles, or explaining it out loud is exactly what surfaces the real issue neither of you had named yet. Either outcome is a win. The only losing move is staying quiet.

A quick framework: name it before you nail it

If you want something more concrete than "trust your gut," try this three step version the next time a review feels off but you can't fully explain why:

Name the feeling, not the fix. Say what you're noticing, not what you think should change. "This feels overly coupled" is enough to open the conversation.
Ask, don't accuse. "What happens if this fails" lands very differently than "this will fail."
Let the conversation do the diagnosis. You don't have to arrive with the answer. The discussion itself usually produces it faster than either person thinking alone.

This works because it separates two things that get merged together way too often: being right, and being useful. You don't have to be right to be useful in a review. You just have to be honest about what you're noticing.

The real cost of staying quiet

Nobody puts "ignored my own intuition" as a root cause in an incident report, but if you read between the lines of enough postmortems, it's there more often than people admit. Someone had a feeling. Someone didn't say it. The system found out the hard way instead.

This isn't a call to nitpick every PR into oblivion or become the reviewer everyone dreads. It's the opposite, actually. It's permission to flag the fuzzy thing without needing to dress it up as a fully formed critique first. Good engineering culture makes room for "I'm not sure, but," not just "here's exactly what's wrong."

So, be honest with yourself for a second. Have you ever sat on a bad feeling about a pull request because you couldn't justify it yet? Or worse, has someone on your team raised one and gotten waved off because it "wasn't specific enough"?

Drop your Jesse Pinkman moment in the comments. The one where you knew, before you knew why.

TL;DR: The feeling that something's off in a code review, even without a clear technical reason, is worth voicing. Silence doesn't make the risk go away, it just delays when the team finds out. Name the feeling, ask instead of accuse, and let the conversation do the rest.

#softwareengineering #codereview #devculture #careeradvice #programming #developerexperience

The Iron Throne Problem: Why Everyone Wants Admin Access and Nobody Should Have It

Tejas Shinkar — Wed, 16 Sep 2026 03:15:46 +0000

Okay real talk — you know that moment in every office when someone says "can you just give me admin access, it'll be faster"?

Yeah. That's the Iron Throne conversation. Every single time.

"Everyone who wants to sit on the Iron Throne, wants to sit on it because it's what they think they need. Nobody sits on it because they should."

Here's the thing nobody says out loud: the person asking for full access almost never actually needs full access. They need one permission, for one task, right now. But "give me admin" is faster to ask for than "give me exactly what I need," so that's what gets asked. And granting it is faster than saying no. So it gets granted.

And then six months later, nobody remembers who has access to what, or why — everyone's just... sitting on thrones they never should've had.

This isn't an AWS thing, or a security-team thing, or even really a tech thing. It's a people thing. Least privilege isn't a compliance checkbox — it's just... not handing out kingdoms because someone asked nicely.

Why "just this once" never stays "just this once"

You've seen this play out before, even if you never called it the Iron Throne. Someone needs to fix one thing urgently, so they get temporary access. The fix works. Nobody remembers to take the access back. Three months later that "temporary" grant is just... permanent, invisible, and nobody's job to notice.

It's not laziness, exactly. It's that revoking access requires someone to actively decide "you don't need this anymore," and that decision has a social cost that granting access never had. Saying yes is free. Saying "actually, give that back" feels like an accusation.

"The King eats last." — except in most orgs, the person who granted the access eats first, and everyone downstream inherits the mess.

The part that actually costs you later

Every access grant is a tiny bit of trust you can't easily take back. Nobody wants to be "that person" who revokes access — it feels petty, like you're accusing someone of something. So it just... stays. Forever. Until an audit, or worse, an incident, forces the question nobody wanted to ask: wait, why does the intern have prod access?

"Power resides where men believe it resides." — and access resides wherever nobody bothered to clean it up.

This is the part that never makes it into onboarding docs. Nobody sits you down on day one and says "hey, by the way, every permission you're granted outlives its usefulness unless someone actively kills it." So permissions just accumulate, quietly, like sediment. And the org chart of "who can technically do what" ends up looking nothing like the org chart of "who's actually supposed to be doing what."

So what actually works?

Not some grand security overhaul. Not a 40-page access policy nobody reads. Just one habit, repeated constantly: ask "what's the smallest thing that solves this?" before "what's the easiest thing to grant?"

That's it. That's the whole philosophy. It's less exciting than a security framework, and it works better than most of them, because it doesn't rely on anyone remembering to clean up later. It just... never lets the mess start.

The uncomfortable truth is that most of us have been on both sides of this. You've asked for more access than you needed because asking twice felt annoying. You've granted more than someone needed because saying "let me scope this properly" felt like friction nobody had time for. Neither of those decisions feels dangerous in the moment. They only feel dangerous in hindsight, usually during an incident review, usually at a time nobody wanted to be having that conversation.

So — be honest. Have you ever been handed the Iron Throne of some system just because it was easier than figuring out what you actually needed? Or worse, have you been the one handing it out?

#softwareengineering #devops #careeradvice #techculture

Server Data Archival to Amazon S3

Tejas Shinkar — Tue, 15 Sep 2026 02:03:24 +0000

Building a production-oriented archival workflow with Python, EC2, IAM and S3

A common infrastructure problem sounds simple:

What should happen to data when an old server is about to be decommissioned?

Deleting the server isn't the difficult part. The difficult part is making sure the right data is preserved, the migration is verified, and the process can be safely repeated without creating duplicates or silently losing files.

That became the problem I decided to solve.

1. The Problem

Imagine a legacy server with a /data directory containing years of application data across logs/, reports/, application/, and backups/.

Before decommissioning it, we need to:

Identify which files are still within the retention period
Archive eligible files
Skip older files
Preserve the directory structure
Upload them securely to Amazon S3
Verify that uploads actually succeeded
Record what happened
Handle failures
Safely rerun the process without duplicating data

Instead of treating this as an S3 upload exercise, I approached it as an infrastructure automation problem.

2. Breaking the Problem Down

I divided the work into three sections.

Section 1 — Script Development

First, I focused on the automation itself. The Python script needed to follow a clear pipeline:

Scan → validate → check retention → check existing archive → upload → verify → log

This gave us a clear separation between the business logic and the AWS infrastructure.

Section 2 — Server Setup

Next, I created a simulated legacy server using an Amazon Linux EC2 instance. Instead of simply running everything as root, I created a /data directory for the source server data, /opt/server-archival for the application, /var/log/server-archival for application logs, a dedicated archiver system user, and a Python virtual environment.

The goal was to make the environment behave more like something we'd actually encounter on a managed server.

Section 3 — Migration & Hardening

Once the script and server were ready, I connected the environment to S3. The EC2 instance received an IAM role scoped to only s3:PutObject and s3:GetObject — no AWS credentials were hardcoded into the Python script.

The final workflow became:

Legacy EC2
    │
    │ /data
    ▼
Python Archival Script
    │
    ├── Retention Check
    ├── Idempotency Check
    ├── Upload
    ├── Verification
    └── Logging
    │
    │ IAM Role
    ▼
Amazon S3
    │
    └── archive/
         ├── logs/
         ├── reports/
         └── ...

3. Retention Logic

The requirement was to archive files within the last 5 years. Initially, I considered simply subtracting 5 × 365 days, but that isn't completely calendar-accurate because of leap years — so the implementation was changed to use calendar-aware date calculation.

This also gave us a chance to test the boundary condition:

File just inside the 5-year window → Archive
File just outside the window → Skip

That kind of boundary testing is easy to overlook in a basic lab.

4. Preserving the File Structure

S3 doesn't have traditional directories — instead, object keys are used. For example, /data/reports/report.csv became s3://legacy-server-archive-bucket/archive/reports/report.csv. The archive/ prefix gives us a clean separation between the original server path and the S3 archive.

5. Making the Process Idempotent

This was one of the most important parts of the project. A migration script shouldn't blindly upload the same file every time it runs, so before uploading, the script checks whether the corresponding S3 object already exists:

Does the object already exist? Yes → skip. No → upload.

This makes the workflow idempotent — if the script crashes halfway through and we run it again, already archived files don't need to be uploaded again.

6. Verification

Uploading successfully isn't enough. After the upload, the script performs an S3 HeadObject check to verify that the object exists:

Not just: upload → assume success. Instead: upload → verify → log success/failure.

That makes the workflow safer for an actual migration scenario.

7. Logging

The script maintains structured logs in a timestamp | level | message format:

INFO | /data/reports/report.csv → Uploaded and verified
INFO | /data/backups/backup.tar.gz → Skip
INFO | /data/logs/app.log → Already archived, skipping upload

At the end, it generates a summary covering files scanned, eligible, uploaded, already archived, skipped, and failed — making the execution auditable instead of relying only on terminal output.

8. Problems We Actually Faced

This project became much more valuable because things didn't always work on the first attempt.

Amazon Linux vs Ubuntu. The initial bootstrap approach used apt, but the EC2 instance was Amazon Linux, so the setup had to be adapted to use dnf.

Python runtime. Boto3 raised a runtime-support warning with the original Python version. Instead of ignoring it, I upgraded the environment to Python 3.11 and recreated only the virtual environment — the application, data, and logs remained separate.

The S3 403 problem. This was probably the most interesting AWS issue. Our IAM policy intentionally didn't include s3:ListBucket. When checking whether an object existed, S3 returned 403 AccessDenied instead of the expected 404 Not Found. At first this looked like an error in the idempotency logic, but it turned into an important AWS permissions lesson: S3's response to a missing object can depend on what permissions the caller has. We adapted the check accordingly without simply adding broader permissions, preserving the least-privilege design.

9. Testing With Realistic Load

Instead of testing with only 2–3 files, I created a larger test dataset — 57 files spread across different directories, file ages, eligible files, old files, and already-archived files.

Metric	First Run	Second Run
Files scanned	57	57
Eligible	33	33
Uploaded	30	0
Already archived	3	33
Skipped	24	24
Failed	0	0

This gave us actual evidence that the idempotency mechanism worked.

10. Production-Oriented Practices

Throughout the project, I deliberately tried to move beyond simply making the code "work."

🔐 Least-privilege IAM — only the required S3 object permissions were granted.
🔑 No credentials in code — the EC2 IAM role provided temporary AWS credentials through the instance role.
👤 Dedicated service user — the archival application didn't need to run as root.
🔄 Idempotency — repeated execution doesn't create duplicate uploads.
🛡️ Validation — the server path is validated before processing.
🔎 Verification — every successful upload is checked.
♻️ Retry handling — Boto3 was configured with standard retry behavior.
📝 Structured logging — execution results and failures are recorded.
🧪 Failure & boundary testing — the workflow was tested against different file ages, permissions, and AWS behavior.
🏃 Dry-run capability — the script can identify what would be archived without actually uploading it.

11. What I Took Away

The biggest learning wasn't how to upload a file to S3. It was learning to think about a cloud problem as a system:

What happens if it fails?
What happens if I run it twice?
What permissions does it really need?
How do I verify the result?
How do I know what happened after the process finishes?

Those questions changed the project from a simple AWS exercise into a production-oriented cloud automation workflow.

Final Architecture

Legacy Server (EC2)
       │
       │ /data
       ▼
Python Archival Automation
       │
       ├── Validate
       ├── Retention Filter
       ├── Idempotency
       ├── Upload
       ├── Verification
       └── Logging
       │
       ▼
IAM Role
(Least Privilege)
       │
       ▼
Amazon S3
       │
       └── archive/

Tech stack: AWS EC2 · Amazon S3 · IAM · Python · Boto3 · Linux · Cloud Automation

The project is available on GitHub as server-data-archival, including the implementation, IAM policy, documentation, and test evidence.

The Easy Part of Coding Got Automated. The Judgment Part Didn't.

Tejas Shinkar — Fri, 11 Sep 2026 12:55:31 +0000

I Thought Coding Was the Hard Part. Then AI Started Writing It For Me.

Asked Claude to write me a Lambda function. Done in 10 seconds.
Asked it which IAM permissions it actually needed. Silence.

So I read the generated code myself. It had asked for s3:, dynamodb:, and a wildcard resource. All Claude did was give the function room to work, not room to fail safely, but I'm the one who has to decide the difference.

Turns out the easy part got automated. The judgment part didn't.
I spent longer scoping the IAM policy down to specific actions and resources than the AI spent writing the whole function.

That's the actual shift nobody's meme-ing about: the code got cheap, review got expensive. Writing was never the bottleneck for anyone who was good at their job. Knowing what "good" looks like still is.

What's one thing AI still can't decide for you?

AWS Route 53 Hands-On Labs

Tejas Shinkar — Wed, 09 Sep 2026 07:53:29 +0000

This practical set covered the core Route 53 DNS and routing concepts through progressively more realistic scenarios.

For the concepts behind these practicals, see the Route 53 session: AWS Route 53: DNS Fundamentals, Hosted Zones, Routing Policies & Resolvers

Practical 1 - Public Hosted Zone + A Record

Objective

Understand a public hosted zone, create an A record, point it to an EC2 public IP, and understand TTL behavior.

What We Did

Created a Public Hosted Zone: route53-lab.example
Route 53 automatically created the NS and SOA records.
Created an A record: www.route53-lab.example → EC2 public IP, TTL 60 seconds, Routing Simple.
Used Route 53 → Test record and confirmed No Error, with the response resolving to the EC2 public IP.
Changed the A record's value and tested again to confirm the update took effect.

Result

The Route 53 record was successfully created and tested.

Important Learning

TTL controls how long a DNS resolver may cache an answer, it does not mean Route 53 itself takes that amount of time to update. Because no real domain was owned or delegated, actual public-browser resolution and real-world TTL propagation could not be tested.

Cleanup

Deleted the custom A record first, then deleted the hosted zone.

Practical 2 - Private Hosted Zone

Objective

Create internal DNS that resolves within an associated VPC.

What We Did

Created a Private Hosted Zone: internal.local
Associated it with the VPC containing the EC2 instance.
Created payment.internal.local → 10.200.1.45.
SSH'd into the EC2 and ran:

nslookup payment.internal.local

Verified the response:

Server:  172.31.0.2
Address: 172.31.0.2#53

Name:    payment.internal.local
Address: 10.200.1.45

Result

Private DNS resolution worked successfully; 172.31.0.2 is the VPC DNS resolver, and it correctly returned 10.200.1.45 as the answer for the record, using the Private Hosted Zone.

Cleanup

Deleted the custom A record and then deleted the private hosted zone.

Practical 3 - Split-Horizon DNS

Objective

Understand how Public and Private Hosted Zones with the same domain can provide different DNS answers.

What We Built

Created both a Public and a Private Hosted Zone using the same domain, split.local - the Private Hosted Zone was associated with the EC2's VPC.

Public www.split.local → 13.127.147.3
Private www.split.local → 172.31.12.251

Test

From the EC2 inside the VPC:

nslookup www.split.local

Name:    www.split.local
Address: 172.31.12.251

The private answer won for the query originating inside the associated VPC. The private A record was then removed and the query repeated - the result was NXDOMAIN, not a fallback to the public zone. This confirmed the private hosted zone continued to take precedence for VPC-originated queries; it simply doesn't have a fallback path to the public zone.

Result

Split-horizon behavior was demonstrated successfully.

Inside the associated VPC → the Private Hosted Zone answers. From public DNS → the Public Hosted Zone answers.

The public side wasn't independently verified from outside the VPC, since the test domain wasn't a real delegated domain.

Cleanup

Deleted the custom records and both hosted zones.

Practical 4 - Weighted Routing / Canary Deployment

Objective

Understand weighted DNS routing and how it can support canary-style releases.

What We Built

Created a Public Hosted Zone (canary-lab.example) with two A records sharing the same name, app.canary-lab.example, both with TTL 60 and Routing Weighted:

Record	Value	Weight	Record ID
Stable	`13.127.147.3`	90	`stable`
Canary	`1.2.3.4`	10	`canary`

Test

Used Route 53 → Test record repeatedly. 13.127.147.3 came back most of the time, while 1.2.3.4 appeared much less frequently - roughly consistent with the 90/10 split, though not an exact 1-in-10 guarantee on any given batch of queries.

Result

Weighted routing behavior was demonstrated. In production, this pattern supports gradual rollout:

90% old version / 10% new version → monitor → 70/30 → 50/50 → 10/90 → 0/100

Important Distinction

Route 53 weighted routing chooses the endpoint returned in the DNS response — it is not an application-layer proxy and doesn't replace an ALB.

Cleanup

Deleted the weighted records and hosted zone.

Practical 5 - Failover Routing + Health Check

Objective

Configure a primary/secondary DNS failover setup and verify that Route 53 serves the secondary endpoint when the primary becomes unhealthy.

What We Built

Two EC2 instances (Primary and Secondary), each returning a distinct response body (PRIMARY SERVER / SECONDARY SERVER) so the active endpoint could be identified from the response alone.

A Route 53 health check (primary-ec2-health) was created: HTTP, port 80, path /, 30-second interval, failure threshold 3.

A Public Hosted Zone (failover-lab.example) was created with app.failover-lab.example configured as:

Primary - A record, Failover = PRIMARY, → 13.127.147.3, health check attached, TTL 60
Secondary - A record, Failover = SECONDARY, → 65.0.61.115, TTL 60

Test

With the primary stopped/unhealthy, the Route 53 Test Record result changed to 65.0.61.115, demonstrating the failover path:

Primary unhealthy → Route 53 → Secondary

Result

The primary-to-secondary failover behavior was successfully demonstrated.

Troubleshooting Observation

When the Primary EC2 was started again, the Route 53 health check continued to show Unhealthy, with health-checker details reporting Connection timed out from multiple health-checker locations. Recovery from Secondary back to Primary was therefore not successfully verified in this lab — the issue was connectivity to the public HTTP endpoint, not the failover record configuration itself, which had already demonstrated the secondary path correctly.

Important Learning

Failover routing depends on endpoint health, not merely whether an EC2 instance is powered on. Health-check detection combined with DNS caching/TTL also means client-visible failover isn't necessarily instantaneous.

Cleanup

Deleted the primary/secondary failover records, the hosted zone, the Route 53 health check, and the Secondary EC2. The original EC2 used for other labs was retained.

Practical 6 - Resolver Endpoints (Conceptual)

This practical was kept conceptual rather than deploying Route 53 Resolver endpoints.

Scenario A - On-Premises → AWS Private Hosted Zone

An on-premises server needs to resolve app.internal, where the name exists in an AWS Private Hosted Zone:

On-premises server → on-prem DNS → AWS Resolver Inbound Endpoint → VPC DNS / Private Hosted Zone → AWS private resource

The Inbound Resolver Endpoint receives DNS queries from outside the VPC.

Scenario B — EC2 → On-Premises DNS

An EC2 instance needs to resolve server.company.local, where the authoritative DNS server exists on-premises:

EC2 → VPC DNS → AWS Resolver Outbound Endpoint → network connection to on-premises DNS → on-prem DNS server → DNS response

Key Distinction

Inbound endpoint → DNS queries coming into AWS/VPC from external networks.

Outbound endpoint → DNS queries going out from AWS/VPC to external DNS servers.

Route 53 Core Mental Model

Simple routing: one record → one/simple answer.

Weighted routing: same name, multiple records, weighted distribution — useful for canary releases, gradual migration, and traffic splitting.

Failover routing: primary + health check → healthy routes to Primary, unhealthy routes to Secondary — useful for high availability and disaster recovery.

Private Hosted Zone: VPC → Private Hosted Zone → internal DNS names — useful for internal services, private application endpoints, and service discovery patterns.

Split-Horizon DNS: the same domain name resolves differently depending on whether the query comes from the public view or the private VPC view — useful whenever the same hostname needs to answer differently based on where the requester is.

Core interview mental model:

Public Hosted Zone → Internet-facing DNS

Private Hosted Zone → VPC/internal DNS

Weighted → distribute DNS answers

Failover → choose Primary while healthy, Secondary when unhealthy

Inbound Resolver → queries into AWS

Outbound Resolver → queries from AWS to external DNS

AWS Observability & Automation — CloudWatch, CloudTrail & EventBridge

Tejas Shinkar — Fri, 04 Sep 2026 12:14:28 +0000

📋 Topics Covered

#	Topic	Type
1	The Big Picture — Why Three Different Services	Concept
2	CloudWatch — Monitoring & Architecture	Concept
3	CloudWatch Metrics — Namespace, Dimensions, Statistic, Period	Concept + Cert
4	High-Resolution Metrics	Concept
5	CloudWatch Logs — Groups, Streams, Events, Retention	Concept
6	CloudWatch Logs Insights	Concept + DevOps
7	CloudWatch Dashboards	Concept
8	CloudWatch Alarms — States, Evaluation, Actions	Concept + Cert
9	CloudWatch Logs → S3 Export	Concept + DevOps
10	Multi-Account and Multi-Region Monitoring	Concept + DevOps
11	EC2 Instance Recovery	Concept + Cert
12	CloudTrail — Management Events	Concept + Interview
13	CloudTrail — Data Events	Concept + Interview
14	CloudTrail Insights	Concept
15	Network Activity Events vs VPC Flow Logs	Concept + Interview
16	EventBridge — Event Bus & Rules	Concept + Cert
17	EventBridge Scheduler	Concept
18	CloudWatch vs CloudTrail vs EventBridge	Interview
19	How the Services Work Together	Concept + DevOps
20	Production Mental Model	DevOps
21	Practice Tasks 1–5	Practice

The Big Picture

AWS environments continuously generate operational and audit data. The challenge isn't collecting it — it's turning it into something useful before it turns into an incident:

Collect → Monitor → Store → Analyze → Act

Each of the three services in this post answers a different question, and mixing them up is one of the most common gaps in AWS troubleshooting interviews and real incident response alike:

CloudWatch → How is the system behaving?
CloudTrail → What happened in the AWS account, and who did it?
EventBridge → An event happened — what should happen next?

They're rarely used in isolation. A real incident usually touches all three: CloudWatch tells you something's wrong, CloudTrail tells you what changed, and EventBridge is what automatically reacts the next time it happens.

Amazon CloudWatch

What Is Monitoring?

Scenario: It's 2 AM. Checkout requests on an e-commerce site start failing. Nobody manually noticed — an alarm did, and paged the on-call engineer with the exact metric that crossed its threshold, five minutes before customers started tweeting about it.

That's what monitoring is for: continuously observing systems and applications so abnormal behavior gets detected, investigated, and acted on — ideally before a human notices the hard way. Common signals worth watching include EC2 CPU utilization and network traffic, RDS connections, SQS queue depth, Lambda errors, application error rates, and request latency.

The operational loop behind this: a problem occurs → telemetry is collected → CloudWatch ingests it → the anomaly is detected and investigated → an alarm, automation, or remediation responds. The production goal is never "watch a dashboard all day" — it's catching meaningful changes early and responding to them, ideally with as little human involvement as the situation allows.

CloudWatch Architecture

                    DATA SOURCES
                         │
        ┌────────────────┼─────────────────┐
        │                │                 │
   AWS Resources    Applications      Multiple
   EC2/ECS/RDS/     & Services        Accounts/Regions
   Lambda/S3/etc.
        │                │                 │
        └────────────────┼─────────────────┘
                         ▼
                  AMAZON CLOUDWATCH
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
       Metrics          Logs          Alarms
          │              │              │
          ▼              ▼              ▼
     Dashboards     Logs Insights     Actions
                                     / Integrations
                                          │
                    ┌─────────────────────┼─────────────────┐
                    ▼                     ▼                 ▼
                   SNS                 Lambda          Auto Scaling
                    │
                    ▼
               Notifications

A useful mental model to keep in mind for everything that follows: metrics measure behavior, logs describe events and details, alarms detect a condition, and actions respond to that condition.

CloudWatch Metrics

Scenario: Your monitoring dashboard shows CPUUtilization climbing from 8% at 10:00, to 11% at 10:01, to 37% at 10:02, to 82% at 10:03 — a metric is exactly that: a numerical measurement observed over time, and that climb is what your alarm will eventually key off of.

Common AWS metrics worth knowing by heart: EC2's CPUUtilization, NetworkIn/NetworkOut, and disk I/O; SQS's ApproximateNumberOfMessagesVisible; Lambda's Invocations, Errors, and Duration; and RDS's CPUUtilization and DatabaseConnections.

Every metric is identified by a combination of parts, and understanding each one is what separates "I set up a metric" from "I understand what I'm actually alarming on":

Namespace — groups related metrics, e.g. AWS/EC2, AWS/SQS, AWS/Lambda, AWS/RDS
Metric name — what's being measured, e.g. CPUUtilization, Errors, Duration
Dimensions — identify the specific resource, e.g. InstanceId=i-0123456789
Statistic — how CloudWatch summarizes the raw values: Average, Minimum, Maximum, Sum, SampleCount
Period — the time interval per datapoint, e.g. 1 minute, 5 minutes, 1 hour

Put together, a complete metric reference looks like: namespace AWS/EC2, metric CPUUtilization, dimension InstanceId=i-0123456789.

The period choice genuinely changes alarm behavior — CPU > 80% for 1 minute will fire far more readily than CPU > 80% for 30 minutes. Alarm design should match the failure condition that actually matters to the business, not just "make the number go away."

High-Resolution Metrics

CloudWatch also supports high-resolution custom metrics for workloads that need finer-grained monitoring than the standard 1-minute granularity. The production decision here isn't "higher resolution is always better" — it's "use the smallest monitoring interval that's actually useful for the problem." Higher resolution helps with fast-changing systems, but it increases telemetry volume and cost, so it should be a deliberate choice, not a default.

CloudWatch Logs

Scenario: A metric shows Lambda Errors = 37 — useful, but it doesn't tell you why. The corresponding log line does: ERROR Database connection timeout, requestId=..., service=order-service. That's the core distinction — a metric tells you what level or how many, a log tells you what exactly happened. For real troubleshooting, you need both.

CloudWatch Logs is organized hierarchically:

/aws/lambda/order-service
    │
    ├── Stream A
    │     ├── Event
    │     ├── Event
    │     └── Event
    │
    └── Stream B
          ├── Event
          └── Event

A Log Group is a logical collection of related logs (e.g. /aws/lambda/order-service). A Log Stream is a sequence of log events from a particular source or execution context within that group. A Log Event is a single entry, like 2026-09-04 10:31:12 ERROR Database timeout.

Log retention shouldn't default to "forever." A production logging strategy decides how long each log group should be kept, weighing troubleshooting needs, compliance, cost, security requirements, and audit requirements — and operational logs often warrant a different retention period than security/audit logs.

CloudWatch Logs Insights

Scenario: An incident just happened, and someone asks "how many AccessDenied errors did we see in the last hour, and from which service?" Scrolling through raw logs isn't an option at scale — this is exactly what Logs Insights is for: querying and aggregating large volumes of log data to turn raw logs into searchable operational evidence.

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

Typical questions it answers well: which requests generated 500 errors, how many AccessDenied errors happened, which service is producing the most failures, and what happened during a specific incident window.

CloudWatch Dashboards

A dashboard provides a centralized operational view. A production application dashboard might track request rate, latency, 5xx errors, EC2 CPU, RDS connections, SQS queue depth, and Lambda errors — but the goal should be decision-making, not visualizing everything you possibly can. A genuinely useful dashboard answers one clear question: can users currently use the application normally?

CloudWatch Alarms

Scenario: CPUUtilization > 80% is set as the alarm condition. At 30% CPU, the alarm sits in OK. At 92% CPU, it flips to ALARM. If there isn't yet enough usable data — a brand-new alarm, or a metric that hasn't produced enough datapoints — it shows INSUFFICIENT_DATA instead of guessing.

The basic flow: a metric is evaluated against a condition → the alarm enters a state → that state triggers an action.

Alarm evaluation settings matter more than they look. Behavior depends on the metric, statistic, period, threshold, evaluation periods, and datapoints-to-alarm — because production alerting should avoid turning every temporary spike into an unnecessary page. CPU > 80% for 1 datapoint is far more trigger-happy than CPU > 80% for several consecutive evaluation periods. Good alerting detects meaningful failure conditions, not just unusual-looking numbers.

Alarms and actions are two separate concerns. A CloudWatch alarm can trigger SNS notifications, Auto Scaling actions, EC2 recovery, or other supported integrations — for example, CPU > threshold → CloudWatch Alarm → SNS → Email/other subscribers. The alarm makes the decision; SNS (or whatever the target is) handles distribution. Keeping that distinction clear avoids a lot of confused debugging later ("the alarm fired but nobody got notified" is almost always an SNS/target problem, not an alarm problem).

CloudWatch Logs → S3 Export

CloudWatch Logs is built for active operations and investigation; S3 is built for durable long-term storage and archival. A logging design should intentionally decide what stays "hot" in CloudWatch versus what gets exported to S3 (CloudWatch Logs → export/archive → Amazon S3) — common reasons include long-term retention, compliance, archival, external analytics, and central data storage.

Multi-Account and Multi-Region Monitoring

Scenario: Your organization runs Production, Development, Staging, and Security in separate AWS accounts, spread across two Regions. Asking an operator to individually check each account during an incident doesn't scale.

Account A ─────┐
Account B ─────┤
Account C ─────┼──→ Central Monitoring Account
Region 1 ──────┤
Region 2 ──────┘
                         │
                         ▼
                 Dashboards / Alarms

A centralized monitoring approach provides visibility from a single monitoring account instead — this is the basis of centralized observability at organizational scale.

EC2 Instance Recovery

EC2 can be configured to recover automatically from certain underlying system/host failures using a CloudWatch alarm — commonly based on StatusCheckFailed_System, which flows StatusCheckFailed_System → CloudWatch Alarm → EC2 Recovery.

The important distinction: instance recovery targets underlying host/system problems, not arbitrary application failures. An underlying host issue is exactly what recovery is for — but Apache crashing, or an application bug, is a completely different failure mode that instance recovery does nothing to fix.

AWS CloudTrail

What Is CloudTrail?

Scenario: Someone deletes an S3 bucket that shouldn't have been deletable. CloudWatch won't tell you who did it — that's not what it's for. CloudTrail is: it answers what API call happened, who performed it, when it happened, which resource was involved, and what request context was recorded.

The cleanest way to keep CloudWatch and CloudTrail straight: CloudWatch asks "how is the system behaving?" CloudTrail asks "what activity happened in AWS?" CloudTrail is therefore heavily used for auditing, security investigation, change tracking, incident investigation, governance, and compliance.

CloudTrail Management Events

Management events cover control-plane operations that change or inspect AWS resources — things like CreateBucket, RunInstances, CreateRole, AttachRolePolicy, CreateSubnet, or DeleteSecurityGroup. Mental model: management events are mainly about AWS resource and configuration activity. If the question is "who changed this security configuration?", CloudTrail's management events are the natural place to look.

CloudTrail Data Events

Data events focus on activity against supported resources' actual data — for S3, that means things like GetObject, PutObject, and DeleteObject. The key distinction: a management event asks "what happened to the AWS resource/configuration?" while a data event asks "what happened to the underlying data?" Data events can be high-volume and are configured separately from management events for exactly that reason.

CloudTrail Insights

Scenario: Your S3 bucket normally sees about 10 DeleteObject calls a minute. Suddenly it's 2,000 a minute. That kind of deviation from a learned baseline is exactly what CloudTrail Insights is built to surface. Mental model: CloudTrail records activity; CloudTrail Insights detects unusual behavior within that activity.

Network Activity Events vs VPC Flow Logs

CloudTrail also supports network activity event logging for supported AWS network-related operations — but don't confuse this with VPC Flow Logs. They are not the same thing: CloudTrail covers service/account activity, while Flow Logs cover network traffic metadata. If you need to know "who called this API," that's CloudTrail. If you need to know "what packets moved between these two hosts," that's Flow Logs.

Amazon EventBridge

What Is EventBridge?

Scenario: An EC2 instance terminates, and instead of someone noticing it's gone twenty minutes later, an automated notification hits the operations team's Slack the moment it happens — no polling, no cron job checking instance state every few minutes. That's the value EventBridge adds: it's primarily an event routing and automation service.

Its basic model: an event arrives on an event bus, a rule examines the event and decides whether it matches, and if it does, the event is routed to a target. Events can originate from AWS services, your own applications, custom applications, or supported SaaS integrations.

Event Bus

                          Event Bus
                    ┌────────┼────────┐
                    ▼        ▼        ▼
                 Rule A   Rule B   Rule C
                    │        │        │
                    ▼        ▼        ▼
                 Lambda     SNS      SQS

The event bus is where events land and get routed. This is what keeps producers and consumers loosely coupled — the service emitting the event doesn't need to know or care what eventually consumes it.

EventBridge Rules

A rule means: when an event matches this pattern, take this action. For example, a rule that watches for EC2 terminations:

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["terminated"]
  }
}

The resulting flow: EC2 terminated → EventBridge → rule matches → SNS / Lambda / SQS. This is a genuinely production-relevant event-driven automation pattern — the exact one behind the "someone gets notified automatically" scenario above.

EventBridge Scheduler

Scheduler answers a fundamentally different question than a rule does: not "when an event happens, react," but "when should something happen?" Instead of waiting on an event trigger (Event happens → EventBridge Rule), you schedule the action directly (2:00 AM every day → EventBridge Scheduler → Lambda). Common uses: running nightly cleanup, generating a morning report, invoking a Lambda periodically, or triggering a one-time task at a specific moment.

The key distinction to keep straight: event-driven means something happens, then react; time-driven means a scheduled time arrives, then execute.

CloudWatch vs CloudTrail vs EventBridge

Service	Primary Question	Typical Use
CloudWatch	How is my system behaving?	Metrics, logs, alarms, dashboards
CloudTrail	What AWS activity happened?	API auditing, investigation, governance
EventBridge	What should happen after an event?	Routing, automation, event-driven workflows

A handful of quick scenario-to-service mappings worth having memorized:

EC2 CPU is too high → CloudWatch
Someone deleted an S3 object → CloudTrail
EC2 termination should notify an operations team → EventBridge → SNS
CPU breach should send an email → CloudWatch Alarm → SNS → Email
Every night at 2 AM, run cleanup → EventBridge Scheduler → Lambda

How the Services Work Together

Real production systems rarely use just one of these — they're stitched together:

                  Application
                      │
             ┌────────┼─────────┐
             ▼        ▼         ▼
          Metrics    Logs    API Activity
             │        │         │
             ▼        ▼         ▼
         CloudWatch CloudWatch CloudTrail
             │        │
             ▼        ▼
           Alarm   Logs Insights
             │
             ▼
            SNS


CloudTrail / AWS event
          ↓
      EventBridge
          ↓
     Lambda / SNS / SQS

This creates a continuous operational loop: observe → detect → investigate → decide → act.

Production Mental Model

Scenario: An incident just started. Instead of randomly clicking through the AWS Console, work through these five questions in order:

What is the system doing? → CloudWatch metrics/logs
What exactly happened in the AWS environment? → CloudTrail
What should automatically happen because of an event? → EventBridge
How should people or systems be notified? → SNS / other targets
What remediation should occur? → Auto Scaling / Lambda / Systems Manager / recovery actions

That maps to a practical investigation flow:

                Incident
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
   CloudWatch   CloudTrail  EventBridge
   telemetry     audit       automation
        │          │          │
        └──────────┼──────────┘
                   ▼
             Investigation
                   │
                   ▼
                Action

Key Takeaways

CloudWatch is the core operational monitoring and observability service: metrics, logs, dashboards, alarms, and related actions.
Metrics show numerical system behavior; logs provide detailed event-level context.
Log Groups → Log Streams → Log Events is the basic CloudWatch Logs hierarchy.
Logs Insights is for querying and aggregating large volumes of log data.
Alarms evaluate metric conditions and can trigger actions such as SNS notifications, scaling, or recovery.
CloudTrail records AWS account/API activity and is central to auditing and investigation.
Management Events focus on control-plane activity; Data Events capture supported data-plane activity such as S3 object operations.
CloudTrail Insights helps detect unusual API activity against established baselines.
EventBridge routes events to targets and enables event-driven automation.
EventBridge Scheduler is for time-driven execution.
Production observability often combines all three services rather than choosing only one.

Remember the three questions:

CloudWatch — How is it behaving?

CloudTrail — What happened?

EventBridge — What should happen next?

Practice Tasks

Practice Task 1 — Custom Metric, Alarm & Notification

Publish a custom CloudWatch metric from an application (or a simple script) representing something business-relevant — e.g. failed login attempts or order-processing time — and configure an alarm on it that notifies an SNS topic when the threshold is breached. Pick a period and evaluation-periods combination deliberately, and be able to justify why it won't fire on a harmless single-datapoint spike.

Practice Task 2 — Investigate an Incident with Logs Insights

Given a Lambda function producing a realistic mix of successful and failing invocations, write Logs Insights queries to answer: how many errors occurred in the last hour, which error message is most frequent, and what the request pattern looked like in the five minutes immediately before the first failure. Produce a short summary of what the queries reveal, as if handing it to an incident commander.

Practice Task 3 — Trace a Change with CloudTrail

Make a deliberate, traceable change in your account — modify a security group rule, delete an S3 object, or alter an IAM policy — then use CloudTrail to answer who made the change, exactly when, from what source IP, and using which API call. Separately, identify whether the change would show up as a management event or a data event, and explain why.

Practice Task 4 — Event-Driven Automation with EventBridge

Build an EventBridge rule that reacts to a real AWS event (e.g. an EC2 instance state change or an S3 object creation) and routes it to a target of your choice (SNS, Lambda, or SQS). Trigger the underlying event for real, and confirm the target actually received and processed it — not just that the rule matched in the console.

Practice Task 5 — Full-Stack Incident Response Scenario

Simulate a small production incident end to end: use CloudWatch to detect a threshold breach (e.g. elevated errors or CPU), use CloudTrail to confirm whether any recent account change correlates with the timing of the incident, and use EventBridge to automatically trigger a remediation action (e.g. invoking a Lambda function or notifying a team) the moment the CloudWatch alarm changes state. Document the full chain — metric → alarm → event → automation — as a single incident timeline.

AWS SNS + SQS Hands-On Lab

Tejas Shinkar — Thu, 03 Sep 2026 12:28:47 +0000

Two practical exercises covering SNS fan-out with SQS/DLQ, and CloudWatch alarm notifications delivered through SNS.

For the concepts behind these practicals, see: AWS SQS + SNS — Decoupling Services with Queues, Topics & Fan-Out Patterns

SNS → SQS Fan-Out with Retry and Dead-Letter Queue

Objective

Build an order-event messaging flow where SNS publishes an event to an SQS queue, then deliberately leave a message unprocessed to observe SQS retries and its eventual move to a dead-letter queue.

Architecture

Publisher
   │
   ▼
SNS Topic
 order-events
   │
   ▼
SQS: order-processing-queue
   │
   ├── processed successfully → deleted
   │
   └── maxReceiveCount (3) exceeded
             │
             ▼
     SQS DLQ: order-processing-dlq

Resources created:

SNS Topic: order-events
SQS main queue: order-processing-queue — Standard type, 30s visibility timeout, 4-day retention, 20s receive wait time, DLQ order-processing-dlq, maxReceiveCount = 3
SQS DLQ: order-processing-dlq

Step 1 — Build the Queue and Topic

Created the DLQ (order-processing-dlq), then the main queue (order-processing-queue) configured to use that DLQ with maxReceiveCount = 3. Created the SNS topic (order-events) and subscribed the main queue to it using the SQS subscription protocol. Verified the queue policy allowed SNS to deliver messages: principal sns.amazonaws.com, action sqs:SendMessage, restricted to source ARN order-events — so only that specific topic can publish into the queue.

Step 2 — Fan-Out Test

Published a test order event to SNS:

{
  "orderId": "ORD-1001",
  "customer": "Tejas",
  "amount": 2499,
  "status": "created"
}

The message appeared in order-processing-queue, confirming SNS → SQS delivery.

Step 3 — Receive-Count / DLQ Test

The message was polled repeatedly without being deleted, to observe the retry-to-DLQ behavior:

Receive #1 → not deleted → visibility timeout expires → Receive #2 → not deleted → Receive #3 → maxReceiveCount reached → message moved to DLQ

The message then appeared in order-processing-dlq, showing a receive count of 4 once polled there — that count reflects its full receive history on the main queue plus the DLQ, not that SNS published it four times.

Result

SNS → SQS delivery confirmed. SQS receive → not deleted → retried → maxReceiveCount reached → moved to DLQ, confirmed end to end.

Key Practical Takeaways

Receiving an SQS message does not automatically delete it.
A consumer should delete a message only after successful processing.
Visibility timeout gives the consumer time to complete processing.
Repeated unsuccessful processing can be isolated in a DLQ.
SNS decouples the publisher from downstream consumers.
The SQS queue policy can restrict which SNS topic can send messages.

CloudWatch CPU Alarm → SNS Email Notifications

Objective

Create a CloudWatch alarm that monitors EC2 CPU utilization and sends an SNS notification to multiple confirmed email subscriptions when the threshold is exceeded.

Architecture

EC2 Instance
   │ CPUUtilization metric
   ▼
CloudWatch Alarm
 ec2-high-cpu-alert
   │ threshold breached → ALARM
   ▼
SNS Topic: cloudwatch-alerts
   │
   ├── Email 1 (confirmed)
   └── Email 2 (confirmed)

Resources created:

SNS Topic: cloudwatch-alerts, with two confirmed email subscriptions
CloudWatch Alarm: ec2-high-cpu-alert — metric CPUUtilization, statistic Average, 1-minute period, condition > 10%

Step 1 — Verify SNS Delivery Independently

Before involving CloudWatch, a direct test message was published straight to cloudwatch-alerts. Both confirmed email addresses received it, proving SNS delivery worked on its own — so any later failure could be isolated to the alarm side rather than the notification side.

Step 2 — Trigger the Alarm

The alarm started in OK once CloudWatch had sufficient data. CPU load was then generated and later stopped:

# Generate load
for i in {1..4}; do yes > /dev/null & done

# Stop load
pkill yes

CPU crossed the 10% threshold and the alarm state changed to ALARM.

Result

EC2 CPU > 10% → CloudWatch Alarm → ALARM → SNS cloudwatch-alerts → Email 1 ✅ → Email 2 ✅

Both confirmed subscriptions received the notification, verifying the full path from metric breach to alarm state change to SNS fan-out to multiple recipients.

Key Practical Takeaways

SNS works well as a notification fan-out layer for operational alerts.
Testing SNS separately helps isolate notification-delivery issues from alarm configuration.
CloudWatch alarms depend on evaluated metric datapoints before changing state.
A controlled CPU-load test is enough to validate the alarm end-to-end.

Practical Completion

Practical	Result
SNS → SQS fan-out	✅ Verified
SQS retry / visibility behavior	✅ Verified
SQS → DLQ after repeated receives	✅ Verified
CloudWatch CPU alarm	✅ Verified
CloudWatch → SNS notification	✅ Verified
Multiple email recipients	✅ Verified

AWS Lambda Hands-On Lab

Tejas Shinkar — Wed, 02 Sep 2026 14:25:17 +0000

⚡ S3 → Lambda Image Resize Pipeline

An event-driven image-processing practical using Amazon S3 and AWS Lambda — verified end-to-end with CloudWatch, then deliberately broken at the IAM layer to practice diagnosing a real production-style failure.

Objective

Build a pipeline where an image uploaded to S3 automatically triggers a Lambda function, which reads the image, resizes it, and writes the generated thumbnail back to a separate S3 prefix — then validate both the working path and a deliberate failure path:

Normal operation: S3 upload → Lambda → resize → thumbnail

Failure handling: S3 upload → Lambda → AccessDenied → CloudWatch diagnosis

Architecture

                    S3 Bucket
        s3-image-resize-lambda-lab
                 /          \
                /            \
        uploads/            thumbnails/
            │                    ▲
            │ ObjectCreated      │ PutObject
            ▼                    │
        ┌────────────────────────────┐
        │          Lambda            │
        │    s3-image-processor      │
        │                            │
        │  1. Read image from S3     │
        │  2. Resize with Pillow     │
        │  3. Write thumbnail to S3  │
        └────────────────────────────┘
                     │
                     ▼
                CloudWatch Logs

The S3 trigger was scoped to the uploads/ prefix only, and Lambda writes its output to thumbnails/ — keeping the generated thumbnail from ever re-triggering the function.

Resources created:

S3 bucket: s3-image-resize-lambda-lab, with prefixes uploads/ (input) and thumbnails/ (output)
Lambda: s3-image-processor — Python 3.12, x86_64
Lambda Layer: pillow-image-processing — Pillow wasn't in the base Python runtime, so it was added as a custom layer
Trigger: S3 ObjectCreated events on uploads/
Execution role: s3:GetObject on uploads/*, s3:PutObject on thumbnails/* — deliberately scoped rather than broad bucket access

Step 1 — Build the Pipeline

The S3 trigger was configured for all object-create events, scoped to the uploads/ prefix with no suffix filter (event type s3:ObjectCreated:*), which also auto-granted S3 the permission to invoke the Lambda. The execution role was given exactly two permissions — s3:GetObject on arn:aws:s3:::s3-image-resize-lambda-lab/uploads/* and s3:PutObject on arn:aws:s3:::s3-image-resize-lambda-lab/thumbnails/* — intentionally avoiding a broader S3 grant.

The handler logic follows a straightforward sequence:

Read S3 event → extract bucket + object key → get the original image from S3 → open with Pillow → resize while preserving aspect ratio → swap uploads/ for thumbnails/ in the key → put the resized image back into S3

The thumbnail target is derived directly from the source key — uploads/photo.png becomes thumbnails/photo.png.

Step 2 — Verify the Normal Workflow

An image was uploaded to uploads/. Lambda was invoked automatically, and the resized thumbnail appeared under thumbnails/ moments later. CloudWatch confirmed the full round trip:

New file uploaded: s3://s3-image-resize-lambda-lab/uploads/56291ef0-ccd4-4201-8334-1915f49fd835.png

Thumbnail created: s3://s3-image-resize-lambda-lab/thumbnails/56291ef0-ccd4-4201-8334-1915f49fd835.png

Every stage of the path checked out: S3 upload, the S3 → Lambda trigger, Lambda execution, the image read, the resize, and the thumbnail write.

Step 3 — Break It on Purpose, Then Diagnose

To simulate a real permissions failure, the s3:GetObject statement for uploads/* was removed from the Lambda execution role (leaving s3:PutObject on thumbnails/* untouched), and a new image was uploaded to uploads/.

The S3 event still fired and Lambda still ran — it just failed partway through. CloudWatch isolated exactly where:

[ERROR] AccessDenied
An error occurred (AccessDenied) when calling the GetObject operation:
User: arn:aws:sts::...:assumed-role/s3-image-processor-role-.../s3-image-processor
is not authorized to perform: s3:GetObject

This pinned the failure precisely on the execution role's permissions, not the S3 trigger, the Lambda Layer, or the runtime — the trigger fired correctly and the function only failed once its code tried to read the source image. The original s3:GetObject grant was then restored and the successful flow re-verified before cleaning up the lab environment.

The event path worked; the execution permission did not.

Troubleshooting Pattern

The sequence that actually isolated the fault, worth reusing on any event-driven pipeline:

Did the upload happen? → did S3 invoke Lambda? → did Lambda start? → where did Lambda fail? → check the CloudWatch error → check the execution-role permission → restore only the required permission

This is more useful than simply checking whether the final thumbnail exists, because it separates an event/configuration problem from an execution/permission problem instead of treating "nothing happened" as one undifferentiated failure.

Key Takeaways

Separate input and output prefixes (uploads/ vs thumbnails/) stop the pipeline's own output from re-triggering itself — for larger systems, separate input/output buckets give an even stronger isolation boundary.
Scope Lambda permissions to the exact paths needed (GetObject on uploads/*, PutObject on thumbnails/*) rather than granting broad, bucket-wide S3 access.
CloudWatch is the first place to look when an event-driven pipeline fails — a successful S3 upload says nothing about whether downstream processing succeeded, since the failure can occur well after the event reaches Lambda.
Event-driven systems fail at multiple independent boundaries, and a production troubleshooting process should identify which boundary failed before changing anything:

S3 upload → S3 event → Lambda invocation → IAM authorization → S3 read → image processing → S3 write

Final Result

Normal path: image upload → S3 /uploads/ → ObjectCreated event → Lambda → Pillow resize → S3 /thumbnails/ → thumbnail ✅

Failure path: image upload → S3 /uploads/ → ObjectCreated event → Lambda → GetObject → AccessDenied ❌ → CloudWatch identifies the missing permission

Outcome: successfully built and verified an S3-triggered Lambda image-processing pipeline end to end, then deliberately broke and correctly diagnosed an IAM permission failure using CloudWatch logs.

So… If AI Can Write Code, What Are Software Engineers Getting Paid For? 👀

Tejas Shinkar — Sun, 30 Aug 2026 16:17:23 +0000

A few years ago, being the person who could turn a requirement into working code was a pretty valuable skill.

Now you can give an AI:

“Build me a REST API with authentication, tests and documentation.”

…and get something surprisingly usable in minutes.

So naturally, the uncomfortable question is:

If AI can write the code, what exactly are software engineers getting paid for?

Here's where it gets interesting.

Imagine an AI-generated feature gets merged.

The tests pass.

The deployment succeeds.

And two hours later, production starts throwing errors.

Now someone has to figure out:

Why only certain users are affected?

Why did latency suddenly spike?

Is the application broken, or is the database struggling?

Did the new code introduce a race condition?

Is the fix actually safe to deploy?

AI can help investigate all of this.

But somebody still has to understand the system well enough to know whether the answer makes sense.

And developers themselves seem to know this.

Stack Overflow's 2025 survey found that 84% of developers use or plan to use AI tools, yet 46% said they don't trust the accuracy of AI output. The biggest frustration? AI solutions that are almost right — with 45% saying debugging AI-generated code can take more time.

There's another interesting signal: developers are much less willing to hand AI high-responsibility work like deployment and monitoring.

Maybe that's the shift.

The valuable engineer isn't necessarily the person who can type the most code anymore.

It's the person who can:

understand the problem → design the system → use AI effectively → question its output → debug what breaks → make the right trade-off.

AI might be making code cheaper.

But good engineering judgment?

That's still expensive.

And maybe that's what we're actually getting paid for. 👀

I Thought Knowing AWS Was The Goal. I Was Wrong.

Tejas Shinkar — Sat, 29 Aug 2026 16:27:31 +0000

When I started learning AWS, I had this very simple idea in my head: learn the services, finish the modules, build a few projects, and eventually I’d be “good at cloud.” So I started collecting names — EC2, S3, IAM, VPC, EFS, CloudFront, Route 53, and a growing graveyard of AWS console tabs. 😂 At first, it felt productive because every completed service felt like another box checked. But somewhere during the networking labs, that started falling apart. I realized I could tell you what a NAT Gateway does, but still stop and ask myself, “Wait… why are we creating another subnet for this?”

That question actually changed how I started learning. Instead of asking, “What does this AWS service do?”, I started asking, “What problem are we solving that requires this thing?” Suddenly VPC started making more sense. A route table wasn't just another AWS resource; it was an answer to “Where should this traffic go?” A NAT Gateway wasn't something you create because a tutorial told you to; it existed because a private machine needed controlled outbound internet access without becoming publicly reachable. Even NACLs became interesting when I deliberately broke one and watched an otherwise healthy request just… hang. 💀

And I started noticing the same thing outside networking. IAM isn't really about memorizing policies — it's about deciding who should be allowed to do what, and from where. CloudFront isn't just “AWS CDN” — it's figuring out which responses are safe to reuse, which ones are personalized, and what happens when your cache thinks two requests are the same when they aren't. Docker isn't impressive because you know docker build; it's useful when you understand what should actually live inside a container, what should stay outside it, and how that behaves when the container gets replaced.

I'm still nowhere near the point where I'd call myself a production DevOps engineer. Most of what I'm learning right now comes from courses, labs, breaking things, fixing them, and trying to reason about what these systems would look like in a real environment. And honestly, that's probably been more valuable than simply finishing lectures. Because the more I learn, the more I realize that companies probably don't need someone who can recite 100 AWS services from memory. They need someone who can look at a broken system and ask the right next question.

Maybe that's the difference between learning tools and learning engineering. One teaches you what exists. The other teaches you why it exists, when to use it, and what happens when it breaks.

I'm curious about people who already work in Cloud/DevOps/SRE:

What was the first thing you learned in production that made you realize, “Yeah… courses really didn't prepare me for this”?

I Thought Caching Was Just About Speed. Then I Found the Security Trap. 💀

Tejas Shinkar — Fri, 28 Aug 2026 05:20:01 +0000

I used to think caching was a pretty simple performance trick: store something closer to the user, serve it faster, reduce the load on the backend, and everyone wins. Then I started digging into CloudFront cache keys and realized caching can become a data-isolation problem if one small decision is wrong. Imagine an application with a GET /profile endpoint where User A sends Cookie: session=userA and gets a response like “Hello Alice, your orders: 12, your balance: ₹42,000.” CloudFront can cache that response so future requests don't always have to reach the origin. Sounds great, right? Until another user requests the same URL and the cache doesn't distinguish the thing that actually makes the response personal.

Now User B sends GET /profile with Cookie: session=userB, and the origin would normally return completely different information such as “Hello Bob, your orders: 4, your balance: ₹8,000.” But if the cache key only considers /profile and ignores the user-specific part of the request, CloudFront can see the request as equivalent to the object it already has. In that situation, User B could receive the response cached for User A. Nothing has crashed, the EC2 instance can be healthy, the database can be healthy, and CloudFront can even be behaving exactly according to its configuration. The actual problem is the caching strategy. That's when caching stops being only a performance optimization and becomes a correctness and security concern.

Now compare that with a public product catalogue. A request like GET /products?id=101 might return the same product information for every user: iPhone 17, ₹79,999, In Stock. That's a fantastic candidate for caching because thousands of users can safely reuse the same response. We want something like one cache object serving many requests, not a separate cache entry for every visitor. But this is where another subtle problem appears: if every request contains an analytics cookie such as analytics_id=83A91, analytics_id=72B42, or analytics_id=91K17, and those values don't change the product response, putting them into the cache key can create thousands or even millions of unnecessary cache variants. The application may still be correct, but cache reuse gets worse, cache efficiency drops, and more requests may end up going back to the origin.

That leads to one of the most useful CloudFront distinctions I've learned: Cache Policy and Origin Request Policy are not answering the same question. Cache Policy is essentially asking, “What makes this request a different cached object?” while Origin Request Policy is asking, “What information does the origin need to receive?” For example, an analytics cookie might need to reach the origin for logging or processing, while still not needing to create a unique cache entry for every user. The important part is to understand which request values actually change the response and which ones are simply metadata. A good cache design therefore doesn't blindly include every cookie, header, or query string just because it is available; it deliberately chooses only the dimensions that matter.

The production lesson for me is that a high cache-hit ratio is not automatically a good outcome. A cache that is extremely fast but returns the wrong user's data is a disaster, not an optimization. The real question isn't “How much can I cache?” but “What can I safely reuse, what makes the response different, and how do I make CloudFront understand that difference?” This also changes how I think about CDN architecture: cache design affects performance, origin load, freshness, correctness, and potentially security at the same time. Static assets such as versioned JavaScript, CSS, images, and public product data are usually much easier to cache aggressively, while personalized responses need a much more careful strategy. Caching is not just about speed — it's about knowing exactly when two requests are allowed to share the same answer.