Part of my AWS learning journey — transitioning from Systems Engineer to Cloud/DevOps. This session moves from networking into the data layer — how AWS manages databases for you, and how caching keeps applications fast.
📋 Topics Covered
| # | Topic | Type |
|---|---|---|
| 1 | Why Separate Compute from Database | Concept |
| 2 | Structured vs Semi-Structured vs Unstructured Data | Concept + Interview |
| 3 | What RDS Actually Is Under the Hood | Concept + Interview |
| 4 | RDS Deployment Options — Single-AZ vs Multi-AZ Instance vs Multi-AZ Cluster | Concept + Cert |
| 5 | Self-Managed DB vs RDS — What AWS Automates | Concept + Interview |
| 6 | Multi-AZ Failover — Step by Step | Concept + Cert |
| 7 | Why RDS Uses a DNS Endpoint, Not an IP | Interview |
| 8 | Amazon Aurora | Concept + Cert |
| 9 | Sharding — Horizontal Scaling | Concept + Interview |
| 10 | Amazon ElastiCache | Concept + Interview |
| 11 | Lab — Creating an RDS Database | Lab |
| 12 | Interview Questions | Interview |
| 13 | Assignment | Practice |
Why Separate Compute from Database
An application server (EC2, Lambda) is built to process data quickly — run logic, transform data, respond to requests. A database is built to store data reliably — persist it, keep it searchable, keep it available to multiple clients at once, and survive restarts.
If you put both on the same machine, a crash takes down your entire system — your data and your processing logic disappear together. Keeping them separate means your application servers can scale up, scale down, or crash and restart, while your data stays safe and available on its own dedicated, purpose-built infrastructure.
The core principle: Compute is optimized for processing. A database is optimized for persistent, reliable, searchable, shared storage. Separating them lets each scale independently and lets your application survive server failures without losing data.
Structured vs Semi-Structured vs Unstructured Data
Not all data looks the same, and AWS has a different storage service optimized for each type.
| Data Type | Schema | Example | AWS Service |
|---|---|---|---|
| Structured | Fixed schema — rows and columns defined upfront | Customer table: id, name, email, order_date | Amazon RDS (relational) |
| Semi-structured | Flexible schema — fields can vary per record | JSON, XML — a user profile with optional fields | Amazon DynamoDB (NoSQL) |
| Unstructured | No predefined format at all | Images, videos, PDFs, log files | Amazon S3 |
Why this distinction matters in real architecture: A retail app might use RDS for orders (structured, needs strong consistency), DynamoDB for a shopping cart (semi-structured, needs speed and flexibility), and S3 for product images (unstructured, needs cheap durable storage). Picking the right store for each data type is a core system design skill.
What RDS Actually Is Under the Hood
This mental model makes RDS click immediately: RDS is not a magical new thing — it's an EC2 instance with attached EBS storage, running a database engine, wrapped in an AWS-managed control plane.
Think of it this way: underneath, RDS is still a virtual machine with a disk, just like any EC2 setup you could build yourself. What AWS adds on top is the automation layer — automatic provisioning, backups, patching, monitoring, scaling, and failover — so you never have to SSH in and manage the database server yourself.
What you get with RDS that you'd have to build yourself with self-managed EC2:
| Task | Self-Managed on EC2 | Amazon RDS |
|---|---|---|
| Install DB engine | Manual | Pre-configured, just choose the engine |
| OS/DB patching | Manual, scheduled downtime | Automated maintenance windows |
| Backups | You script and schedule them | Automated daily snapshots + point-in-time recovery |
| Failover | You build and test it yourself | Automatic (Multi-AZ) |
| Monitoring | You set up CloudWatch agents | Built-in metrics out of the box |
| Scaling storage | Manual resize + downtime | Can auto-scale storage |
RDS Deployment Options
RDS offers three deployment models, and this is one of the most tested concepts in the SAA-C03 exam. The image from class (RDS Console's deployment screen) shows this exact choice.
Memory trick:
- Single-AZ → One copy 🏠
- Multi-AZ Instance → One primary + one standby 🏠🏠
- Multi-AZ DB Cluster → One writer + multiple readers 🏠🏠🏠
Side-by-Side Comparison
| Single-AZ | Multi-AZ DB Instance | Multi-AZ DB Cluster | |
|---|---|---|---|
| Instances | 1 | 2 (1 primary + 1 standby) | 3 (1 writer + 2 readable standbys) |
| Uptime SLA | 99.5% | 99.95% | 99.95% |
| Automatic failover | ❌ No | ✅ Yes | ✅ Yes |
| Standby readable | N/A | ❌ No (standby is idle, failover-only) | ✅ Yes (readers serve read traffic) |
| Read scaling | ❌ No | ❌ No | ✅ Yes |
| Redundancy across AZs | ❌ No | ✅ Yes | ✅ Yes |
| Use case | Dev, test, non-critical workloads | Production, HA required | Production, HA + read-heavy workloads |
| Cost | Lowest | Medium (2x compute) | Highest (3x compute) |
Reading the deployment screenshot from class:
- Multi-AZ DB Cluster (3 instances): Primary instance + SSD in AZ 1, two "Readable standby + SSD" instances in AZ 2 and AZ 3. There's a separate "Reader endpoint" that load-balances read queries across the standbys, while the "Write/read endpoint" always points to the primary.
- Multi-AZ DB Instance (2 instances): Primary in AZ 1, a standby in AZ 2 that has no endpoint of its own — it exists purely as a failover target, not for serving traffic.
- Single-AZ (1 instance): Just the primary, no redundancy at all. If AZ 1 has an issue, there's no automatic failover — you'd be restoring from backup.
🎯 Cert tip: The key differentiator that trips people up — in a Multi-AZ Instance deployment, the standby is NOT readable. You cannot send read queries to it; it exists purely for failover. Only a Multi-AZ DB Cluster gives you readable standbys that also help with read scaling.
Self-Managed DB vs RDS — What AWS Automates
On a self-managed database (say, PostgreSQL installed manually on an EC2 instance), a DBA is responsible for installing the database binaries, configuring database parameters (memory allocation, connection limits, query optimization settings), and managing how clients connect.
RDS automates all three of these:
| Self-Managed Task | RDS Equivalent |
|---|---|
| Manually install DB binaries | Choose an engine (PostgreSQL, MySQL, etc.) — AWS manages the binary |
Manually edit config files (postgresql.conf) |
Parameter Groups — a managed collection of engine settings you can tune without SSH access |
| Manage static connection strings, update on every failover | DNS Endpoint — a stable hostname that AWS keeps pointed at the current primary |
Parameter Groups deserve a specific callout — instead of editing a config file directly on the server, you modify settings through a Parameter Group in the RDS Console, and AWS applies them to the instance (sometimes requiring a reboot, depending on the parameter).
Multi-AZ Failover — Step by Step
This is the mechanism that makes Multi-AZ deployments valuable — understanding exactly what happens during a failure builds real confidence for both interviews and production incidents.
The failover sequence:
Primary Crash → Standby Promoted → DNS Endpoint Updated (TTL = 5 seconds) → Application Resolves New IP → Reconnect → Database AvailableTotal time: roughly 60–120 seconds
Breaking down each step:
- Primary Crash — the primary instance becomes unreachable (hardware failure, AZ outage, or a manual failover for maintenance)
- Standby Promoted — AWS automatically promotes the standby replica in the other AZ to become the new primary
- DNS Endpoint Updated — the RDS DNS endpoint's record is updated to point to the new primary's IP address, with a TTL (Time To Live) of about 5 seconds
- Application Resolves New IP — because the DNS TTL is so short, client applications quickly pick up the new IP on their next DNS lookup
- Reconnect — the application's connection pool reconnects using the newly resolved IP
- Database Available — normal operations resume
The entire process typically completes within 60 to 120 seconds — no manual intervention required.
Why RDS Uses a DNS Endpoint, Not an IP
This is a genuinely good interview question because it tests whether you understand why a design choice was made, not just what it is.
Q: Why does RDS use a DNS endpoint instead of exposing the database IP directly?
During a Multi-AZ failover, the database's IP address changes — the standby gets promoted and it has a different IP than the old primary. If applications connected using a hardcoded IP, every failover would require someone to manually update every application's configuration and restart it — completely defeating the purpose of "automatic" failover.
Instead, applications connect using a stable RDS DNS endpoint (something like
mydb.abc123xyz.ap-south-1.rds.amazonaws.com) that never changes. Behind the scenes, AWS updates what that DNS name resolves to. With a TTL of about 5 seconds, client applications re-resolve the DNS quickly and reconnect to the new primary automatically — with zero configuration changes needed on the application side.
This is exactly the same principle used by Elastic IPs and Route 53 health checks elsewhere in AWS — decouple the stable identifier from the underlying resource so the underlying resource can change without breaking anything upstream.
Amazon Aurora
Amazon Aurora is AWS's own cloud-native relational database, compatible with MySQL and PostgreSQL (meaning your existing MySQL/PostgreSQL drivers and tools work without changes) but built with fundamentally different internal architecture for much higher performance.
What makes Aurora different from standard RDS engines:
| Feature | Standard RDS (MySQL/PostgreSQL) | Aurora |
|---|---|---|
| Compute + Storage | Coupled together | Separated — storage scales independently |
| Replication | You configure it | Automatic, across 3 AZs, 6 copies of data |
| Storage auto-scaling | Manual/limited | Automatic, up to 128 TB |
| Failover speed | ~60-120 seconds | Faster (typically under 30 seconds) |
| Performance | Baseline | Significantly higher throughput |
| Cost | Lower | Higher (premium for the performance) |
The "separates compute from storage" concept, explained simply: In a standard database, if you need more storage, you often need to resize the whole instance. In Aurora, the storage layer is a separate, distributed system that grows automatically as your data grows — the compute instance (which runs the actual query engine) can scale independently. This is why Aurora can offer both faster failover (the storage layer already has 6 copies ready) and larger scale (storage isn't tied to a single disk).
🎯 Cert tip: When a scenario mentions "MySQL-compatible," "PostgreSQL-compatible," "high performance," and "automatic storage scaling" together, the answer is almost always Aurora, not standard RDS.
Sharding — Horizontal Scaling for Databases
As a database grows, at some point one server (even a very large one) can't handle the write load or storage anymore. Sharding is the technique for scaling out — instead of one giant database, you split it into multiple smaller databases called shards, each holding a portion of the data.
How it works: A large database is partitioned into multiple smaller databases (shards) based on some key — for example, splitting users A-M into Shard 1 and users N-Z into Shard 2. Each shard stores only its subset of the data. The application (or a routing layer) determines which shard to query based on the data being requested, so both storage and write load get distributed across multiple database servers instead of one.
Why this matters: Vertical scaling (bigger instance) has a ceiling — eventually you run out of bigger instance types. Sharding is how systems scale writes and storage beyond what any single database server could handle, at the cost of added application complexity (your app needs to know which shard to query).
💡 Where this connects: This is conceptually the same "horizontal vs vertical scaling" idea from the ELB/ASG session — just applied to databases instead of compute. Vertical = bigger box. Horizontal (sharding) = more boxes, each handling a slice of the problem.
Amazon ElastiCache
Even a well-tuned database has a limit to how many reads it can serve per second, and every query — even a fast one — has some latency. ElastiCache is AWS's fully managed in-memory caching service — it stores frequently accessed data in RAM, which is dramatically faster than querying a database on disk.
How caching works in an application:
Application needs data → checks the cache first → if the data is there (cache hit), return it instantly from RAM → if the data isn't there (cache miss), query the database, get the result, store it in the cache for next time, then return it to the caller.
This means the first request for a piece of data is a normal database query, but every subsequent request for the same data is served from memory — orders of magnitude faster, and it takes load off the database entirely.
Three Caching Engines
| Engine | Persistence | Replication | Status |
|---|---|---|---|
| Valkey | ✅ Yes | ✅ Yes | Open-source fork of Redis (community-driven, actively used going forward) |
| Redis OSS | ✅ Yes | ✅ Yes | Long-standing standard, still widely used |
| Memcached | ❌ No | ❌ No | Simpler, but largely fallen out of favor — no persistence or replication means data loss on restart |
💡 Why Valkey exists: After a licensing change to Redis, the open-source community forked the last fully open-source version of Redis into a new project called Valkey — it's Linux Foundation-backed and functionally very similar to Redis. AWS supports it as a first-class ElastiCache engine going forward.
Memcached is essentially legacy at this point — no persistence means a restart wipes your cache entirely, and no replication means no high availability. Almost all new projects choose Redis OSS or Valkey.
🧪 Lab — Creating an RDS Database (Progress So Far)
This is the practical work completed so far in class — continuing in a future session to connect it to EC2 and build the full application.
Step 1 — Create a DB Subnet Group
A DB Subnet Group tells RDS which subnets (across which AZs) it's allowed to place database instances in.
Go to RDS Console → Subnet Groups → Create DB Subnet Group. Choose your existing VPC. Select the private subnets across each of the 3 Availability Zones — databases should never sit in a public subnet.
Step 2 — Create the Database
RDS Console → Create database → chose the PostgreSQL engine → selected the engine version → set a DB instance identifier (a name for this specific database instance) → set the master username → chose self-managed password authentication → selected the DB instance class (Burstable class, e.g.,
db.t3.micro) → chose storage type and allocated storage size → left storage auto-scaling disabled for now.
Step 3 — Connectivity
Chose "Connect to an EC2 compute resource" during setup → this triggered creating an EC2 instance to connect to the database → created an Internet Gateway and attached it to the VPC → configured subnet routes → configured connectivity, including an additional VPC security group (
launch-wizard-2) → disabled Performance Insights for now (cost-saving on a lab account) → DB Subnet Group was auto-selected based on the earlier subnet group creation → clicked Create Database.
What's left for the next session: actually connecting from the EC2 instance to the RDS database, verifying connectivity, and building out the application layer on top.
⚡ Quick Revision
Why Separate Compute & DB
Compute = optimized for processing. Database = optimized for persistent, reliable, shared storage. Separation lets each scale independently and survive failures.
Data Types
Structured (fixed schema) → RDS. Semi-structured (flexible schema, JSON/XML) → DynamoDB. Unstructured (no format) → S3.
What RDS Really Is
EC2 + EBS + DB engine + AWS-managed automation layer (provisioning, backups, patching, monitoring, scaling, failover).
Three Deployment Options
- Single-AZ 🏠 → one instance, no failover, 99.5% SLA
- Multi-AZ Instance 🏠🏠 → primary + non-readable standby, automatic failover, 99.95% SLA
- Multi-AZ Cluster 🏠🏠🏠 → one writer + 2 readable standbys, failover + read scaling, 99.95% SLA
Failover Flow
Primary Crash → Standby Promoted → DNS Updated (TTL 5s) → App Resolves New IP → Reconnect → Available. Total: 60-120 seconds.
Why DNS Endpoint, Not IP
IP changes on failover. DNS endpoint stays constant; AWS updates what it resolves to. Apps never need reconfiguration.
Aurora
MySQL/PostgreSQL-compatible, but compute and storage are separated. 3 AZs, 6 copies of data automatically. Auto-scales storage to 128 TB. Faster failover, higher performance than standard RDS.
Sharding
Horizontal scaling for databases — split one large DB into multiple shards, each holding a subset of data, distributing storage and write load.
ElastiCache
In-memory caching. Cache hit = instant from RAM. Cache miss = query DB, store result in cache, return. Engines: Valkey (Redis fork, active), Redis OSS (standard), Memcached (legacy, no persistence/replication).
💼 Interview Questions
Q1: Why does AWS keep compute and database storage separate instead of running everything on one server?
Compute is optimized for processing logic quickly, while databases are optimized for persistent, reliable, and shared storage. Keeping them separate allows application servers to scale independently, restart, or fail without losing data, since the database lives on its own dedicated, durable infrastructure.
Q2: What is the difference between a Multi-AZ DB Instance and a Multi-AZ DB Cluster?
A Multi-AZ DB Instance has one primary and one standby — the standby is not readable and exists purely for automatic failover. A Multi-AZ DB Cluster has one writer and two readable standbys across different AZs, providing both automatic failover and read scaling, since the standbys can serve read traffic through a separate reader endpoint.
Q3: Walk through what happens during an RDS Multi-AZ failover.
The primary instance becomes unavailable, so AWS automatically promotes the standby in another AZ to primary. The RDS DNS endpoint's record is updated to point to the new primary's IP, with a TTL of about 5 seconds. Applications re-resolve the DNS quickly and reconnect using the new IP — no manual configuration changes needed. The whole process typically takes 60 to 120 seconds.
Q4: Why does RDS use a DNS endpoint instead of a static IP address?
Because the underlying IP address changes during failover — the standby that gets promoted has a different IP than the old primary. If applications connected via a hardcoded IP, every failover would require manual reconfiguration. The DNS endpoint stays constant while AWS updates what it resolves to, with a short TTL so clients pick up the change within seconds automatically.
Q5: What makes Amazon Aurora different from standard RDS engines like MySQL or PostgreSQL on RDS?
Aurora separates compute from storage — the storage layer is a distributed system that automatically replicates across 3 AZs with 6 copies of data and scales up to 128 TB automatically. This architecture also enables faster failover and significantly higher performance compared to standard RDS engines, while remaining compatible with existing MySQL and PostgreSQL tooling.
Q6: What is sharding and when would you use it?
Sharding is a horizontal scaling technique where a large database is split into multiple smaller databases (shards), each storing a subset of the data. It's used when a single database instance can no longer handle the write load or storage requirements — even with vertical scaling — because it distributes both storage and write traffic across multiple servers.
Q7: How does ElastiCache improve application performance?
Applications check the cache before querying the database. On a cache hit, data is returned instantly from RAM, which is dramatically faster than a database query. On a cache miss, the application queries the database, then stores the result in the cache for future requests. This reduces database load and significantly improves response times for frequently accessed data.
Q8: Why is Memcached rarely chosen for new projects compared to Redis OSS or Valkey?
Memcached has no persistence (data is lost on restart) and no replication (no high availability). Redis OSS and Valkey both support persistence and replication, making them suitable for production caching layers where data durability and availability matter — which is why most new projects choose one of those two instead.
🔬 Assignment
- Create a web server on EC2, host a simple application on it, connect it to the RDS database created in this session's lab, build a basic frontend, and create a form that makes an entry into the database.
AWS Session 11 — RDS & ElastiCache | Cloud + DevOps learning journey — Systems Engineer → Cloud/DevOps Engineer
Practical RDS-to-EC2 connection and remaining lab steps to be completed in a follow-up practical session.
Top comments (0)