DEV Community: Joud Awad

Database Internals For System Design

Joud Awad — Mon, 31 Aug 2026 15:56:17 +0000

They added an index and the query got slower.

Then they doubled the instance size and bought back 400 milliseconds.

So they put Redis in front of it. The hit rate came back at 4 percent. Then they added a read replica, and it fell 45 seconds behind.

Four fixes. All reasonable. All wrong.

None of it was bad luck. Every one of those numbers was predictable from what the engine was doing underneath the query.

The index lost because the planner's row estimate was off — and an index scan is two hops, find the entry, then go fetch the row. Past enough rows, reading the pages straight through just wins.

The 4 percent wasn't a Redis problem. It was the diagnosis. A cache is a second buffer pool. If the working set doesn't fit in the first one, it isn't going to fit in the second one either.

And the replica lag came from the same place. Replication is the write-ahead log, shipped. An UPDATE in Postgres never overwrites a row — it writes a new version. More versions, more WAL, more lag.

Same layer, three times. Storage.

Most of us learned databases top-down: write the query, add an index when it's slow, add a replica when the index stops helping. That's backwards. Indexes make no sense without pages. Joins make no sense without indexes. And nobody explains the WAL until the replica is already 45 seconds behind.

So I built the bottom-up version I wish I'd had.

2 hours 22 minutes. Seven layers — storage, indexes, pagination, execution, transactions, read scaling, schema. One query, one 200 million row table, all the way up. Seven checkpoint quizzes with real countdowns, so you answer before you hear the answer.

If you only watch part of it, watch the bottom four.

https://youtu.be/uy2yom0AlhA

DynamoDB Client API Tutorial: Query, Pagination, Cost Explained

Joud Awad — Mon, 24 Aug 2026 09:24:02 +0000

There is no SQL in DynamoDB.

No query language, no syntax to memorize, no planner deciding how your request gets run. You import a client and you call a method.

In Postgres, "give me one row" and "give me a thousand rows" are the same SELECT with different clauses, and the engine works out the rest. In DynamoDB those are different methods and you're the one choosing. GetItem for a single item by its key. BatchGetItem for up to 100 of them. Query for a run of items under one partition key. Scan for when you've given up and want the whole table.

The method name is the operation. That's the shift people miss coming from SQL: you're not composing a statement, you're picking an action. One item or many, atomic or not, read or write. Three decisions, and the entire client API falls out of them.

What you do have to learn is the cost, because it isn't hidden behind a planner any more. 4 KB read blocks, 1 KB write blocks, and a round-up rule that makes a 200-byte item cost a full block. A strongly consistent read costs double an eventual one. A read inside a transaction doubles again. Once you know which method you called, you can price the call before you run it.

So I ran every one of them on camera, from real NestJS code, with the consumed capacity on screen next to each call.

https://youtu.be/JeyZ-kN0H7Y?si=ZkvqVIC1TCjCX06k

DynamoDB Table Design: Complete Data Modeling Guide

Joud Awad — Thu, 20 Aug 2026 16:16:51 +0000

DynamoDB Data modeling is one of the hardest things that appears on paper. Still, once you understand the different access patterns and how DynamoDB queries your data, it starts being an easy task.

DynamoDB uses a unique combination of Partition Key (PK) and sort keys (SK) in order to store and query your data, and understanding how those two works under the hood is the first part of working with DynamoDB, the hard part and the most challenging is to understand how to model your data, how to model your access pattern, and how you can extract questions from your own system and convert them into a working DynamoDB Tables.

In this video, we walk you through a full DynamoDB end-to-end table design, covering all the access patterns DynamoDB uses to build three real-world case examples.

At the end of this video, you will be able to:

Understand how DynamoDB Data Modeling works.
Build your own DynamoDB Tables.
Convert your own use case into a question DynamoDB is able to answer for you.
Model any real-world use case that you may face in your own application.
Master the Single-Table Design in DynamoDB
Work and interact with all the differnet access patterns DynamoDB provide to you.

Video Link: https://www.youtube.com/watch?v=rbP1jGNzG2A

Caching For System Design: Redis, CDN, Cache Patterns Explained

Joud Awad — Mon, 17 Aug 2026 15:53:52 +0000

A 99% cache hit rate means your database has never been load tested
at real traffic.

It has only ever seen 1% of it.

Wikimedia published the postmortem. Their hit ratio fell while demand
stayed flat, and the origin took five times its normal load. Nothing
else changed. The cache just stopped absorbing.

Which means the number on your dashboard is not a performance metric.
It is the size of your outage.

I went through primary sources on caching for a video recently, and
that reframe was the one that stuck. A cache is a second copy of your
data. Every hard question after that is about the copy, not the cache.

Then there is the cost, which most advice treats as settled. Memory
is cheap, databases are expensive, therefore cache. I priced it out
in August 2026. A cache node runs about 21% cheaper than a comparable
read replica per gigabyte. Not an order of magnitude. Twenty-one
percent.

A cache pays for the traffic it absorbs and nothing else. That is the
entire business case.

The patterns everyone teaches do not survive checking either.
Write-through has three incompatible vendor definitions. AWS lists
"never stale" as an advantage. Microsoft ships a repair function
because the cache write can fail after the commit. Write-around, the
fourth column in every caching-patterns table online, appears in none
of the nine primary sources I checked.

And there is one test almost nobody runs. It is a single sentence in
AWS's own guidance: run load tests with caches disabled.

Full breakdown, 65 minutes, chaptered, every claim tied to a primary
source: https://youtu.be/ETvLl-8bPbo

What is inside the box, the six layers a request passes through, the
write patterns, the failure modes, and the cost math.

System Design Crash Course: Scaling 100 to 100M Users

Joud Awad — Mon, 10 Aug 2026 16:11:43 +0000

Most systems don't break at 100 million users.

They break at 5,000, because someone architected for 100 million.

Every scaling stage has exactly one bottleneck. Solve that one. Ignore the rest.

100 users: one server. App, database, static files, all of it. This takes you further than you think, and it's the only stage where you can debug production by reading a single log file.

10,000 users: your database is competing with your app for CPU. Split them onto separate boxes. Nothing else. Not microservices, not Kafka.

100,000 users: one app instance can't hold the traffic. Add a load balancer and a second instance. The moment you do, session state in memory becomes a bug. Move sessions out.

1 million users: reads are drowning your primary database. Read replicas and a cache layer. Most teams ship the cache and skip the invalidation strategy, then spend a quarter chasing stale data.

10 million users: your bottleneck is geographic, not computational. A request from Singapore to us-east-1 spends roughly 200ms in network round trips before your code runs. CDN for static assets, then regional deployments.

100 million users: now you shard. Not before. Sharding trades away joins, cross-entity transactions, and easy schema migrations in exchange for write throughput. It is a permanent decision.

What most scaling content gets wrong:

It presents these as a checklist of technologies to adopt. They aren't. Each stage is a specific failure mode with a specific fix, and applying a stage-six fix at stage two buys you the operational cost of a large system with the traffic of a small one.

I've watched a team run Kubernetes and a service mesh to serve 400 requests per minute. The tooling wasn't wrong. The timing was.

Scale isn't a number you design for. It's a sequence of bottlenecks you meet one at a time.

I put the full progression into a visual walkthrough: the architecture at every stage, and the exact signal that tells you it's time to move to the next one.

https://youtu.be/fwVGulYwlak

Observability Crash Course

Joud Awad — Mon, 03 Aug 2026 15:36:31 +0000

Observability is being able to look inside a running system.

Not at one moment you planned for. At any moment, and understand what it's actually doing.

It takes several components working together to make a system visible from the outside.

This week's deep dive on System Design Labs is a full crash course.

The pillars, how they fit, and how to follow one request end to end across the whole system.

We cover OpenTelemetry, logs, metrics, traces, SLOs, sampling, profiling, eBPF, and more.

https://youtu.be/umm-MyCl3Q4

Day 12/30 AWS System Design Patterns

Joud Awad — Sat, 01 Aug 2026 16:45:23 +0000

A document processing pipeline runs as an AWS Step Functions Standard workflow (serverless workflow orchestrator — Standard tier bills per state transition, not per execution or per second). Seven top-level states: receive document, validate format, extract text, run classification, store result, notify downstream, update audit log. The extract-text state is a Map state (fans out over an array — each item runs its own sub-workflow, and every state inside every iteration is billed as a transition) that processes the document page by page. Documents average 9 pages, and each page runs a 7-state pipeline: render, OCR, confidence check, a retry choice, redaction, store page, emit metrics.

Average execution takes 4–6 seconds end to end.

When the pipeline processed 80,000 documents per month, the Step Functions bill was $140. The platform grew. Last month it processed 2.2 million documents. The bill was $3,850.

Nothing changed in the workflow. No new states were added. The team is deciding whether to optimize or migrate.

What is the correct explanation, and what is the right fix?

A) Step Functions Standard pricing is per execution at a flat rate — 2.2M executions scaled the cost linearly and only reducing volume can reduce it; but Standard Workflows are not priced per execution — the billable unit is the state transition, and a single execution can contain a few transitions or hundreds

B) Standard bills every state transition, and the Map state multiplies them — 7 top-level states plus 9 pages × 7 states per page is roughly 70 transitions per document; at 2.2M documents that is ~154M transitions, and at $0.025 per 1,000 transitions, $3,850; the bill scales with documents × pages, not documents

C) Step Functions Standard includes a per-second execution charge — longer executions at higher volume caused the spike; but duration billing is the Express Workflow model; a Standard execution that sits in a Wait state for 24 hours costs the same transitions as one finishing in 4 seconds

D) The increase is CloudWatch Logs (AWS monitoring, charges per GB ingested) ingestion from execution history — a real cost at scale, but a 27× bill jump from $140 to $3,850 with unchanged logging configuration cannot come from log ingestion

Answer in the comments.

DynamoDB Indexes Deep Dive (GSI vs LSI)

Joud Awad — Thu, 30 Jul 2026 15:41:03 +0000

Your DynamoDB table has plenty of write capacity. Your writes are getting throttled anyway.

Nothing is wrong with your table. The problem is a GSI you added eight months ago and forgot about.

AWS has a name for this: GSI back-pressure. If an index can't keep up with the writes flowing into it, DynamoDB throttles the base table until it catches up. Your table is healthy. Your index is the bottleneck. The error lands on the write you just made.

This confuses people because of one wrong assumption almost everyone carries over from SQL.

In Postgres, an index is a structure that lives next to your table and helps the query planner find rows faster. In DynamoDB, a GSI is a separate physical copy of your data, sitting on different hardware, keyed differently.

Not a pointer. A copy.

Once you actually believe that, every strange behavior stops being strange.

Why can't a GSI give you strong consistency? The data is somewhere else. DynamoDB can't promise you're reading the latest write.

Why does one write cost more than one write? Changing an indexed attribute from A to B is two writes, one to delete the old index entry and one to add the new. Five GSIs covering that attribute means six writes for every one you make. You pay for all six.

Why does an index throttle your table? Because it isn't part of your table. It's a second table you didn't realize you were operating.

Now the part that makes the copy model worth it.

If an item is missing the GSI's key attribute, it never enters the index at all. DynamoDB just skips it.

Picture an orders table with 20 million rows. Maybe 1% are still open and need processing. Put an attribute called is_open on only those, leave it off everything else, and build a GSI keyed on it. Your index holds a few thousand rows instead of 20 million. Queries run in single-digit milliseconds, storage costs almost nothing, and you wrote zero filtering logic.

When an order ships, delete the attribute. DynamoDB removes it from the index for you. The index garbage collects itself as work moves through your system.

One trap before you go build this. If you're using LSIs instead, every item that shares a partition key has to fit in 10 GB, and that includes every LSI copy. Three LSIs means four copies of each item counting against the same ceiling. Cross it and DynamoDB starts rejecting writes for that customer. You can't drop an LSI without rebuilding the whole table.

And if you're still gluing strings together like SHIPPED#us-east-1 to fake a composite key, stop. Since November 2025 a GSI can take up to four partition key attributes and four sort key attributes natively. No synthetic fields, no backfills. Sort keys match strictly left to right, so define them in the order you actually query them.

I put the whole mental model into 25 minutes, including the write sharding pattern for when adaptive capacity can't save you: https://youtu.be/xVsEviV2vNA

Day 9/30 AWS System Design Patterns

Joud Awad — Wed, 29 Jul 2026 17:46:10 +0000

A lending platform runs its loan approval workflow as a Lambda durable function (Lambda's checkpoint-and-replay execution mode — on every resume, the handler re-runs from the top and skips completed durable operations using their stored results). The flow: validate the application (a context.step()), pull the credit report (a step), then a review loop — context.wait() for 6 hours, check whether a human reviewer has approved, repeat for up to 5 days. Function timeout: 90 seconds per invocation. Execution timeout: 7 days.

At the top of the handler — before any step — the code downloads a 40 MB compliance ruleset from S3 (object storage) and parses it. It takes about 50 seconds. The engineer put it there deliberately: "several steps need it, so load it once."

The workflow ships. Applications flow through reviews for two days. Then, gradually, applications stop progressing. By Thursday, 1,900 applications are frozen mid-review. The business logic has thrown zero errors. The durable execution history shows executions alive and well within their 7-day window — just never advancing past their latest wait. The only anomaly: invocation duration on the function is pinned at exactly 90 seconds.

What is happening?

A) The executions exceeded the durable execution timeout — but the execution timeout (7 days here, up to 1 year) is a separate setting from the Lambda function timeout, and the history shows executions only 2–3 days old and still active

B) context.wait() keeps the function running and billing during the 6-hour waits, and the accumulated wait time consumed the timeout — but a wait suspends the execution entirely; each resume is a brand-new invocation with a fresh 90-second function timeout

C) The 50-second ruleset load sits outside any durable operation — so it is not checkpointed, and it re-executes on every single resume; each resume pays 50 seconds of S3 load plus replay of a growing checkpoint log before reaching any new work, and after enough review cycles a resume can no longer finish inside 90 seconds — it is killed mid-replay, retried, and killed again

D) The checkpoint log hit its per-execution size limit and Lambda silently stopped scheduling replays — but exceeding durable execution limits surfaces as explicit errors in the execution history, not a silent stall, and the 90-second duration signature points at the invocation clock

Answer in the comments.

Day 8/30 AWS System Design Patterns

Joud Awad — Tue, 28 Jul 2026 15:44:11 +0000

One malformed record. 61,000 duplicate loyalty credits.

A retail platform streams purchase events into a Kinesis Data Stream (streaming log — consumers track their own position via a checkpoint; the stream never deletes on read). A Lambda function (serverless compute) consumes the stream through an event source mapping (polls the shard and invokes Lambda with batches — advances the checkpoint only when the whole batch succeeds) with a batch size of 100. For each record, the function calls the loyalty service and credits points to the customer's account.

The pipeline has run cleanly for a year at 2 million events per day.

On Tuesday at 9:40 AM, a producer deployment ships a bug: one purchase event is published with a null user_id. It lands at position 73 of a batch. The function processes records 1 through 72 — credits 72 customers — then throws on record 73. The invocation fails.

The event source mapping retries. Not record 73 — the batch. Records 1 through 72 are credited again. Record 73 throws again. Retry. Again. The mapping's retry setting is the default: keep retrying until the record ages out of the stream's 24-hour retention.

Support notices at 1:50 PM: four hours, roughly 850 retry cycles, 61,000 duplicate credits, and every purchase event behind the bad record on that shard is stuck waiting. No Lambda alarm fired — the function's errors look like a routine blip at first glance, because the error count is one per retry, not 61,000.

Why did already-processed records run again, and what is the correct fix?

A) Kinesis (streaming log) delivered the same records multiple times during the incident — but a stream is not a delivery service that can duplicate; the consumer reads from a position it controls, and the stream served exactly what was asked for

B) The checkpoint advances per batch, not per record — a thrown error rewinds the consumer to the last committed checkpoint, so every record after it, including the 72 that succeeded, re-executes on every retry until the batch finally succeeds or the poison record expires

C) Two concurrent Lambda invocations processed the same shard and raced — but the event source mapping invokes one batch at a time per shard precisely to preserve ordering; there is no concurrent second reader on the shard

D) The loyalty service retried the credits internally — but it received separate, fully-formed requests carrying no shared idempotency key; from its side these were 850 distinct instructions to credit points, and it executed them correctly

Answer in the comments.

Day 7/30 AWS System Design Patterns

Joud Awad — Mon, 27 Jul 2026 20:40:47 +0000

A multi-tenant SaaS platform sends all background notifications — email, SMS, webhook calls — through a single SQS Standard queue (message queue — a message is removed only when a consumer explicitly deletes it or its retention period expires, 4 days by default). The queue feeds a Lambda function (serverless compute) through an event source mapping (polls the queue, invokes Lambda with batches, and deletes messages when the invocation succeeds). The team did things right: the queue has a Dead Letter Queue (DLQ — a second queue that receives messages after they fail delivery more than maxReceiveCount times) with maxReceiveCount of 5, and an alarm on DLQ depth.

Friday, 6 PM: a routine deployment ships two changes. One updates the email provider SDK — and breaks the provider credential lookup. The other is a "hardening" change from code review: the handler body is wrapped in a try/catch that logs any exception and returns normally, "so one bad notification can't poison the batch."

Saturday, 9 AM: support is flooded. No customer received anything overnight — 41,000 notifications gone. The team checks CloudWatch (AWS monitoring service): zero function errors all night. Queue depth: zero. The DLQ: empty. The DLQ alarm never fired. Every dashboard is green.

Where did 41,000 messages go?

A) The messages crossed an internal receive-count threshold and SQS deleted them — but no such mechanism exists; receive count only routes messages to a DLQ via a configured redrive policy, and SQS deletes nothing based on how many times a message was received

B) The handler caught every exception and returned success — Lambda reported a clean invocation, so the event source mapping did what success means: it called DeleteMessage on every message in the batch; the messages were deleted legitimately, one green invocation at a time

C) The queue's retention period expired overnight — but retention defaults to 4 days; messages a few hours old cannot age out, and expiry would not explain the zero-error, empty-DLQ picture either way

D) The deployment disabled the event source mapping, so messages were never consumed — but an unconsumed queue shows growing depth; a queue at zero depth with no errors means messages were received, "processed," and deleted

Answer in the comments.

Redis Cluster Mode Explained

Joud Awad — Mon, 27 Jul 2026 11:10:54 +0000

Your Redis node is out of memory.

You add three more nodes. Still full, and nothing got faster.

You added copies of your data when what you actually needed was room for it.

Replication and sharding solve completely different problems. A replica is a full copy of the primary, which is great for reads and failover, but it buys you zero extra space. Sharding splits the key space so each primary owns a different slice. One cake cut across three plates, not the same cake run through a photocopier.

Cluster mode is how Redis does the slicing: 16,384 hash slots, CRC16 of the key picks the slot, and the slot owns the key, not the node.

What nobody warns you about is that flipping it on changes your application code:

→ MGET, MULTI, and Lua only run when every key hashes to the same slot. Otherwise you get CROSSSLOT.
→ Hash tags like {42} force keys together, and overusing them rebuilds the exact bottleneck you were escaping.
→ Only DB 0 exists. SELECT is gone.
→ Your one giant leaderboard still sits on a single node, because a key cannot be split. No amount of re-sharding fixes that. It's a data model change.

I broke the whole thing down visually: slots, MOVED vs ASK, gossip and failover, the async durability gap, and when ElastiCache's toggle actually earns its keep.

https://youtu.be/0G5_w2lX02o