DEV Community: Joud Awad

CDN For System Design

Joud Awad — Mon, 14 Sep 2026 16:15:23 +0000

Say "add a CDN" in a system design interview and the next four questions decide whether you pass.

One photo answers all four. Uploaded once to a server in Virginia. Opened by someone in Sydney.

Before a single line of your code runs, geography has already charged that viewer about 930 milliseconds. A bigger server fixes none of it.

The four questions, usually within thirty seconds:

What is your cache key? What does a TTL expiry cost the origin? Which edge answers, and why? How do you serve a private file from a public machine?

I followed that one photo for 44 minutes through all four. A few things I wish someone had told me earlier.

Most engineers think the cache key is the URL. It isn't. It's the method plus the URI plus whatever your config or a Vary header bolts on. CloudFront's default CachingOptimized policy ignores query strings entirely. Cloudflare's default includes the full query string, verbatim, order included. Same photo URL, opposite behavior, two vendors.

Add Vary: User-Agent and you get one cache entry per browser build. One extra header can take a hit ratio from the high nineties into single digits.

A long TTL is permission, not a guarantee.

Eviction happens whenever the edge wants the space back. And when a viral object does expire, it expires on every tier at once, so edges everywhere turn around and hit your origin in the same window. That's the thundering herd. Request collapsing and Origin Shield exist so that as few as one request per object reaches you.

Purges are priced by the path you submit, not the files it clears. CloudFront gives you 1,000 free paths a month, then $0.005 each. Which is the actual argument for versioned filenames: photo.a3f9.jpg never needs purging.

Which edge answers? GeoDNS guesses from your resolver's address, not yours. EDNS Client Subnet patches the guess. Anycast removes it: every edge announces the same IP, and failover becomes a routing event with no DNS TTL to wait out.

And the number interviewers ask for is the wrong one. Going from 90% to 95% hit ratio sounds like five points. It halves your origin load. Origin offload is the metric. Hit ratio is the proxy.

The private file? Signed URLs for a single object. Signed cookies when one viewer needs a hundred of them.

Full walkthrough, one photo end to end, built from the vendor docs and the RFCs rather than folklore:

https://www.youtube.com/watch?v=gV_z0uG8_uo

Circuit Breaker Pattern Explained

Joud Awad — Mon, 07 Sep 2026 15:25:36 +0000

Payments went fully down, and checkout was fine.

The next week payments stayed up and answered in 30 seconds instead of 40 milliseconds. The entire site went dark.

A dead dependency is a small problem. A slow one takes everything with it.

Every call still waiting is holding a thread, a database connection and a chunk of memory that the product page needed too. Nothing throws an error. The pool is just gone.

That's the cascading failure a circuit breaker exists to stop — and it's why the breaker is never installed on its own. A request meets a timeout, then a retry policy, then the breaker, then a fallback. In that order.

Most of us built that chain from a tutorial that said "open after 5 consecutive failures."

Go read your actual config.

Resilience4j, Hystrix, Polly and gobreaker don't count consecutive failures. They trip on a failure rate across a sliding window, gated by a minimum call volume. Which is better — until you notice that on a low-traffic endpoint, two of those defaults combine to leave the breaker unable to open at all. Not "slow to open." Unable.

Three more things I had wrong for years:

Retries are selfish. Exponential backoff without jitter doesn't spread the load, it just reschedules the same spike for later.

You can have full throughput and zero goodput at the same time, and every dashboard will look busy while nothing succeeds.

And AWS's own Builders' Library warns that circuit breakers add modal behavior and recovery time — its SDK ships a retry token bucket instead. Amazon has a written case against fallback code, too. The pattern everyone teaches as the answer is the one Amazon argues you should think twice about.

So I built the version that includes the numbers and the counterarguments: 36 minutes, one call from checkout to payments, all the way through.

If you take one thing: an open breaker with no alarm on its state changes is a silent partial outage.

https://youtu.be/rvKKj7JU92g

Database Internals For System Design

Joud Awad — Mon, 31 Aug 2026 15:56:17 +0000

They added an index and the query got slower.

Then they doubled the instance size and bought back 400 milliseconds.

So they put Redis in front of it. The hit rate came back at 4 percent. Then they added a read replica, and it fell 45 seconds behind.

Four fixes. All reasonable. All wrong.

None of it was bad luck. Every one of those numbers was predictable from what the engine was doing underneath the query.

The index lost because the planner's row estimate was off — and an index scan is two hops, find the entry, then go fetch the row. Past enough rows, reading the pages straight through just wins.

The 4 percent wasn't a Redis problem. It was the diagnosis. A cache is a second buffer pool. If the working set doesn't fit in the first one, it isn't going to fit in the second one either.

And the replica lag came from the same place. Replication is the write-ahead log, shipped. An UPDATE in Postgres never overwrites a row — it writes a new version. More versions, more WAL, more lag.

Same layer, three times. Storage.

Most of us learned databases top-down: write the query, add an index when it's slow, add a replica when the index stops helping. That's backwards. Indexes make no sense without pages. Joins make no sense without indexes. And nobody explains the WAL until the replica is already 45 seconds behind.

So I built the bottom-up version I wish I'd had.

2 hours 22 minutes. Seven layers — storage, indexes, pagination, execution, transactions, read scaling, schema. One query, one 200 million row table, all the way up. Seven checkpoint quizzes with real countdowns, so you answer before you hear the answer.

If you only watch part of it, watch the bottom four.

https://youtu.be/uy2yom0AlhA

DynamoDB Client API Tutorial: Query, Pagination, Cost Explained

Joud Awad — Mon, 24 Aug 2026 09:24:02 +0000

There is no SQL in DynamoDB.

No query language, no syntax to memorize, no planner deciding how your request gets run. You import a client and you call a method.

In Postgres, "give me one row" and "give me a thousand rows" are the same SELECT with different clauses, and the engine works out the rest. In DynamoDB those are different methods and you're the one choosing. GetItem for a single item by its key. BatchGetItem for up to 100 of them. Query for a run of items under one partition key. Scan for when you've given up and want the whole table.

The method name is the operation. That's the shift people miss coming from SQL: you're not composing a statement, you're picking an action. One item or many, atomic or not, read or write. Three decisions, and the entire client API falls out of them.

What you do have to learn is the cost, because it isn't hidden behind a planner any more. 4 KB read blocks, 1 KB write blocks, and a round-up rule that makes a 200-byte item cost a full block. A strongly consistent read costs double an eventual one. A read inside a transaction doubles again. Once you know which method you called, you can price the call before you run it.

So I ran every one of them on camera, from real NestJS code, with the consumed capacity on screen next to each call.

https://youtu.be/JeyZ-kN0H7Y?si=ZkvqVIC1TCjCX06k

DynamoDB Table Design: Complete Data Modeling Guide

Joud Awad — Thu, 20 Aug 2026 16:16:51 +0000

DynamoDB Data modeling is one of the hardest things that appears on paper. Still, once you understand the different access patterns and how DynamoDB queries your data, it starts being an easy task.

DynamoDB uses a unique combination of Partition Key (PK) and sort keys (SK) in order to store and query your data, and understanding how those two works under the hood is the first part of working with DynamoDB, the hard part and the most challenging is to understand how to model your data, how to model your access pattern, and how you can extract questions from your own system and convert them into a working DynamoDB Tables.

In this video, we walk you through a full DynamoDB end-to-end table design, covering all the access patterns DynamoDB uses to build three real-world case examples.

At the end of this video, you will be able to:

Understand how DynamoDB Data Modeling works.
Build your own DynamoDB Tables.
Convert your own use case into a question DynamoDB is able to answer for you.
Model any real-world use case that you may face in your own application.
Master the Single-Table Design in DynamoDB
Work and interact with all the differnet access patterns DynamoDB provide to you.

Video Link: https://www.youtube.com/watch?v=rbP1jGNzG2A

Caching For System Design: Redis, CDN, Cache Patterns Explained

Joud Awad — Mon, 17 Aug 2026 15:53:52 +0000

A 99% cache hit rate means your database has never been load tested
at real traffic.

It has only ever seen 1% of it.

Wikimedia published the postmortem. Their hit ratio fell while demand
stayed flat, and the origin took five times its normal load. Nothing
else changed. The cache just stopped absorbing.

Which means the number on your dashboard is not a performance metric.
It is the size of your outage.

I went through primary sources on caching for a video recently, and
that reframe was the one that stuck. A cache is a second copy of your
data. Every hard question after that is about the copy, not the cache.

Then there is the cost, which most advice treats as settled. Memory
is cheap, databases are expensive, therefore cache. I priced it out
in August 2026. A cache node runs about 21% cheaper than a comparable
read replica per gigabyte. Not an order of magnitude. Twenty-one
percent.

A cache pays for the traffic it absorbs and nothing else. That is the
entire business case.

The patterns everyone teaches do not survive checking either.
Write-through has three incompatible vendor definitions. AWS lists
"never stale" as an advantage. Microsoft ships a repair function
because the cache write can fail after the commit. Write-around, the
fourth column in every caching-patterns table online, appears in none
of the nine primary sources I checked.

And there is one test almost nobody runs. It is a single sentence in
AWS's own guidance: run load tests with caches disabled.

Full breakdown, 65 minutes, chaptered, every claim tied to a primary
source: https://youtu.be/ETvLl-8bPbo

What is inside the box, the six layers a request passes through, the
write patterns, the failure modes, and the cost math.

System Design Crash Course: Scaling 100 to 100M Users

Joud Awad — Mon, 10 Aug 2026 16:11:43 +0000

Most systems don't break at 100 million users.

They break at 5,000, because someone architected for 100 million.

Every scaling stage has exactly one bottleneck. Solve that one. Ignore the rest.

100 users: one server. App, database, static files, all of it. This takes you further than you think, and it's the only stage where you can debug production by reading a single log file.

10,000 users: your database is competing with your app for CPU. Split them onto separate boxes. Nothing else. Not microservices, not Kafka.

100,000 users: one app instance can't hold the traffic. Add a load balancer and a second instance. The moment you do, session state in memory becomes a bug. Move sessions out.

1 million users: reads are drowning your primary database. Read replicas and a cache layer. Most teams ship the cache and skip the invalidation strategy, then spend a quarter chasing stale data.

10 million users: your bottleneck is geographic, not computational. A request from Singapore to us-east-1 spends roughly 200ms in network round trips before your code runs. CDN for static assets, then regional deployments.

100 million users: now you shard. Not before. Sharding trades away joins, cross-entity transactions, and easy schema migrations in exchange for write throughput. It is a permanent decision.

What most scaling content gets wrong:

It presents these as a checklist of technologies to adopt. They aren't. Each stage is a specific failure mode with a specific fix, and applying a stage-six fix at stage two buys you the operational cost of a large system with the traffic of a small one.

I've watched a team run Kubernetes and a service mesh to serve 400 requests per minute. The tooling wasn't wrong. The timing was.

Scale isn't a number you design for. It's a sequence of bottlenecks you meet one at a time.

I put the full progression into a visual walkthrough: the architecture at every stage, and the exact signal that tells you it's time to move to the next one.

https://youtu.be/fwVGulYwlak

Observability Crash Course

Joud Awad — Mon, 03 Aug 2026 15:36:31 +0000

Observability is being able to look inside a running system.

Not at one moment you planned for. At any moment, and understand what it's actually doing.

It takes several components working together to make a system visible from the outside.

This week's deep dive on System Design Labs is a full crash course.

The pillars, how they fit, and how to follow one request end to end across the whole system.

We cover OpenTelemetry, logs, metrics, traces, SLOs, sampling, profiling, eBPF, and more.

https://youtu.be/umm-MyCl3Q4

Day 12/30 AWS System Design Patterns

Joud Awad — Sat, 01 Aug 2026 16:45:23 +0000

A document processing pipeline runs as an AWS Step Functions Standard workflow (serverless workflow orchestrator — Standard tier bills per state transition, not per execution or per second). Seven top-level states: receive document, validate format, extract text, run classification, store result, notify downstream, update audit log. The extract-text state is a Map state (fans out over an array — each item runs its own sub-workflow, and every state inside every iteration is billed as a transition) that processes the document page by page. Documents average 9 pages, and each page runs a 7-state pipeline: render, OCR, confidence check, a retry choice, redaction, store page, emit metrics.

Average execution takes 4–6 seconds end to end.

When the pipeline processed 80,000 documents per month, the Step Functions bill was $140. The platform grew. Last month it processed 2.2 million documents. The bill was $3,850.

Nothing changed in the workflow. No new states were added. The team is deciding whether to optimize or migrate.

What is the correct explanation, and what is the right fix?

A) Step Functions Standard pricing is per execution at a flat rate — 2.2M executions scaled the cost linearly and only reducing volume can reduce it; but Standard Workflows are not priced per execution — the billable unit is the state transition, and a single execution can contain a few transitions or hundreds

B) Standard bills every state transition, and the Map state multiplies them — 7 top-level states plus 9 pages × 7 states per page is roughly 70 transitions per document; at 2.2M documents that is ~154M transitions, and at $0.025 per 1,000 transitions, $3,850; the bill scales with documents × pages, not documents

C) Step Functions Standard includes a per-second execution charge — longer executions at higher volume caused the spike; but duration billing is the Express Workflow model; a Standard execution that sits in a Wait state for 24 hours costs the same transitions as one finishing in 4 seconds

D) The increase is CloudWatch Logs (AWS monitoring, charges per GB ingested) ingestion from execution history — a real cost at scale, but a 27× bill jump from $140 to $3,850 with unchanged logging configuration cannot come from log ingestion

Answer in the comments.

DynamoDB Indexes Deep Dive (GSI vs LSI)

Joud Awad — Thu, 30 Jul 2026 15:41:03 +0000

Your DynamoDB table has plenty of write capacity. Your writes are getting throttled anyway.

Nothing is wrong with your table. The problem is a GSI you added eight months ago and forgot about.

AWS has a name for this: GSI back-pressure. If an index can't keep up with the writes flowing into it, DynamoDB throttles the base table until it catches up. Your table is healthy. Your index is the bottleneck. The error lands on the write you just made.

This confuses people because of one wrong assumption almost everyone carries over from SQL.

In Postgres, an index is a structure that lives next to your table and helps the query planner find rows faster. In DynamoDB, a GSI is a separate physical copy of your data, sitting on different hardware, keyed differently.

Not a pointer. A copy.

Once you actually believe that, every strange behavior stops being strange.

Why can't a GSI give you strong consistency? The data is somewhere else. DynamoDB can't promise you're reading the latest write.

Why does one write cost more than one write? Changing an indexed attribute from A to B is two writes, one to delete the old index entry and one to add the new. Five GSIs covering that attribute means six writes for every one you make. You pay for all six.

Why does an index throttle your table? Because it isn't part of your table. It's a second table you didn't realize you were operating.

Now the part that makes the copy model worth it.

If an item is missing the GSI's key attribute, it never enters the index at all. DynamoDB just skips it.

Picture an orders table with 20 million rows. Maybe 1% are still open and need processing. Put an attribute called is_open on only those, leave it off everything else, and build a GSI keyed on it. Your index holds a few thousand rows instead of 20 million. Queries run in single-digit milliseconds, storage costs almost nothing, and you wrote zero filtering logic.

When an order ships, delete the attribute. DynamoDB removes it from the index for you. The index garbage collects itself as work moves through your system.

One trap before you go build this. If you're using LSIs instead, every item that shares a partition key has to fit in 10 GB, and that includes every LSI copy. Three LSIs means four copies of each item counting against the same ceiling. Cross it and DynamoDB starts rejecting writes for that customer. You can't drop an LSI without rebuilding the whole table.

And if you're still gluing strings together like SHIPPED#us-east-1 to fake a composite key, stop. Since November 2025 a GSI can take up to four partition key attributes and four sort key attributes natively. No synthetic fields, no backfills. Sort keys match strictly left to right, so define them in the order you actually query them.

I put the whole mental model into 25 minutes, including the write sharding pattern for when adaptive capacity can't save you: https://youtu.be/xVsEviV2vNA

Day 9/30 AWS System Design Patterns

Joud Awad — Wed, 29 Jul 2026 17:46:10 +0000

A lending platform runs its loan approval workflow as a Lambda durable function (Lambda's checkpoint-and-replay execution mode — on every resume, the handler re-runs from the top and skips completed durable operations using their stored results). The flow: validate the application (a context.step()), pull the credit report (a step), then a review loop — context.wait() for 6 hours, check whether a human reviewer has approved, repeat for up to 5 days. Function timeout: 90 seconds per invocation. Execution timeout: 7 days.

At the top of the handler — before any step — the code downloads a 40 MB compliance ruleset from S3 (object storage) and parses it. It takes about 50 seconds. The engineer put it there deliberately: "several steps need it, so load it once."

The workflow ships. Applications flow through reviews for two days. Then, gradually, applications stop progressing. By Thursday, 1,900 applications are frozen mid-review. The business logic has thrown zero errors. The durable execution history shows executions alive and well within their 7-day window — just never advancing past their latest wait. The only anomaly: invocation duration on the function is pinned at exactly 90 seconds.

What is happening?

A) The executions exceeded the durable execution timeout — but the execution timeout (7 days here, up to 1 year) is a separate setting from the Lambda function timeout, and the history shows executions only 2–3 days old and still active

B) context.wait() keeps the function running and billing during the 6-hour waits, and the accumulated wait time consumed the timeout — but a wait suspends the execution entirely; each resume is a brand-new invocation with a fresh 90-second function timeout

C) The 50-second ruleset load sits outside any durable operation — so it is not checkpointed, and it re-executes on every single resume; each resume pays 50 seconds of S3 load plus replay of a growing checkpoint log before reaching any new work, and after enough review cycles a resume can no longer finish inside 90 seconds — it is killed mid-replay, retried, and killed again

D) The checkpoint log hit its per-execution size limit and Lambda silently stopped scheduling replays — but exceeding durable execution limits surfaces as explicit errors in the execution history, not a silent stall, and the 90-second duration signature points at the invocation clock

Answer in the comments.

Day 8/30 AWS System Design Patterns

Joud Awad — Tue, 28 Jul 2026 15:44:11 +0000

One malformed record. 61,000 duplicate loyalty credits.

A retail platform streams purchase events into a Kinesis Data Stream (streaming log — consumers track their own position via a checkpoint; the stream never deletes on read). A Lambda function (serverless compute) consumes the stream through an event source mapping (polls the shard and invokes Lambda with batches — advances the checkpoint only when the whole batch succeeds) with a batch size of 100. For each record, the function calls the loyalty service and credits points to the customer's account.

The pipeline has run cleanly for a year at 2 million events per day.

On Tuesday at 9:40 AM, a producer deployment ships a bug: one purchase event is published with a null user_id. It lands at position 73 of a batch. The function processes records 1 through 72 — credits 72 customers — then throws on record 73. The invocation fails.

The event source mapping retries. Not record 73 — the batch. Records 1 through 72 are credited again. Record 73 throws again. Retry. Again. The mapping's retry setting is the default: keep retrying until the record ages out of the stream's 24-hour retention.

Support notices at 1:50 PM: four hours, roughly 850 retry cycles, 61,000 duplicate credits, and every purchase event behind the bad record on that shard is stuck waiting. No Lambda alarm fired — the function's errors look like a routine blip at first glance, because the error count is one per retry, not 61,000.

Why did already-processed records run again, and what is the correct fix?

A) Kinesis (streaming log) delivered the same records multiple times during the incident — but a stream is not a delivery service that can duplicate; the consumer reads from a position it controls, and the stream served exactly what was asked for

B) The checkpoint advances per batch, not per record — a thrown error rewinds the consumer to the last committed checkpoint, so every record after it, including the 72 that succeeded, re-executes on every retry until the batch finally succeeds or the poison record expires

C) Two concurrent Lambda invocations processed the same shard and raced — but the event source mapping invokes one batch at a time per shard precisely to preserve ordering; there is no concurrent second reader on the shard

D) The loyalty service retried the credits internally — but it received separate, fully-formed requests carrying no shared idempotency key; from its side these were 850 distinct instructions to credit points, and it executed them correctly

Answer in the comments.