DEV Community: Joud Awad

System Design Crash Course: Scaling 100 to 100M Users

Joud Awad — Mon, 10 Aug 2026 16:11:43 +0000

Most systems don't break at 100 million users.

They break at 5,000, because someone architected for 100 million.

Every scaling stage has exactly one bottleneck. Solve that one. Ignore the rest.

100 users: one server. App, database, static files, all of it. This takes you further than you think, and it's the only stage where you can debug production by reading a single log file.

10,000 users: your database is competing with your app for CPU. Split them onto separate boxes. Nothing else. Not microservices, not Kafka.

100,000 users: one app instance can't hold the traffic. Add a load balancer and a second instance. The moment you do, session state in memory becomes a bug. Move sessions out.

1 million users: reads are drowning your primary database. Read replicas and a cache layer. Most teams ship the cache and skip the invalidation strategy, then spend a quarter chasing stale data.

10 million users: your bottleneck is geographic, not computational. A request from Singapore to us-east-1 spends roughly 200ms in network round trips before your code runs. CDN for static assets, then regional deployments.

100 million users: now you shard. Not before. Sharding trades away joins, cross-entity transactions, and easy schema migrations in exchange for write throughput. It is a permanent decision.

What most scaling content gets wrong:

It presents these as a checklist of technologies to adopt. They aren't. Each stage is a specific failure mode with a specific fix, and applying a stage-six fix at stage two buys you the operational cost of a large system with the traffic of a small one.

I've watched a team run Kubernetes and a service mesh to serve 400 requests per minute. The tooling wasn't wrong. The timing was.

Scale isn't a number you design for. It's a sequence of bottlenecks you meet one at a time.

I put the full progression into a visual walkthrough: the architecture at every stage, and the exact signal that tells you it's time to move to the next one.

https://youtu.be/fwVGulYwlak

Observability Crash Course

Joud Awad — Mon, 03 Aug 2026 15:36:31 +0000

Observability is being able to look inside a running system.

Not at one moment you planned for. At any moment, and understand what it's actually doing.

It takes several components working together to make a system visible from the outside.

This week's deep dive on System Design Labs is a full crash course.

The pillars, how they fit, and how to follow one request end to end across the whole system.

We cover OpenTelemetry, logs, metrics, traces, SLOs, sampling, profiling, eBPF, and more.

https://youtu.be/umm-MyCl3Q4

Day 12/30 AWS System Design Patterns

Joud Awad — Sat, 01 Aug 2026 16:45:23 +0000

A document processing pipeline runs as an AWS Step Functions Standard workflow (serverless workflow orchestrator — Standard tier bills per state transition, not per execution or per second). Seven top-level states: receive document, validate format, extract text, run classification, store result, notify downstream, update audit log. The extract-text state is a Map state (fans out over an array — each item runs its own sub-workflow, and every state inside every iteration is billed as a transition) that processes the document page by page. Documents average 9 pages, and each page runs a 7-state pipeline: render, OCR, confidence check, a retry choice, redaction, store page, emit metrics.

Average execution takes 4–6 seconds end to end.

When the pipeline processed 80,000 documents per month, the Step Functions bill was $140. The platform grew. Last month it processed 2.2 million documents. The bill was $3,850.

Nothing changed in the workflow. No new states were added. The team is deciding whether to optimize or migrate.

What is the correct explanation, and what is the right fix?

A) Step Functions Standard pricing is per execution at a flat rate — 2.2M executions scaled the cost linearly and only reducing volume can reduce it; but Standard Workflows are not priced per execution — the billable unit is the state transition, and a single execution can contain a few transitions or hundreds

B) Standard bills every state transition, and the Map state multiplies them — 7 top-level states plus 9 pages × 7 states per page is roughly 70 transitions per document; at 2.2M documents that is ~154M transitions, and at $0.025 per 1,000 transitions, $3,850; the bill scales with documents × pages, not documents

C) Step Functions Standard includes a per-second execution charge — longer executions at higher volume caused the spike; but duration billing is the Express Workflow model; a Standard execution that sits in a Wait state for 24 hours costs the same transitions as one finishing in 4 seconds

D) The increase is CloudWatch Logs (AWS monitoring, charges per GB ingested) ingestion from execution history — a real cost at scale, but a 27× bill jump from $140 to $3,850 with unchanged logging configuration cannot come from log ingestion

Answer in the comments.

DynamoDB Indexes Deep Dive (GSI vs LSI)

Joud Awad — Thu, 30 Jul 2026 15:41:03 +0000

Your DynamoDB table has plenty of write capacity. Your writes are getting throttled anyway.

Nothing is wrong with your table. The problem is a GSI you added eight months ago and forgot about.

AWS has a name for this: GSI back-pressure. If an index can't keep up with the writes flowing into it, DynamoDB throttles the base table until it catches up. Your table is healthy. Your index is the bottleneck. The error lands on the write you just made.

This confuses people because of one wrong assumption almost everyone carries over from SQL.

In Postgres, an index is a structure that lives next to your table and helps the query planner find rows faster. In DynamoDB, a GSI is a separate physical copy of your data, sitting on different hardware, keyed differently.

Not a pointer. A copy.

Once you actually believe that, every strange behavior stops being strange.

Why can't a GSI give you strong consistency? The data is somewhere else. DynamoDB can't promise you're reading the latest write.

Why does one write cost more than one write? Changing an indexed attribute from A to B is two writes, one to delete the old index entry and one to add the new. Five GSIs covering that attribute means six writes for every one you make. You pay for all six.

Why does an index throttle your table? Because it isn't part of your table. It's a second table you didn't realize you were operating.

Now the part that makes the copy model worth it.

If an item is missing the GSI's key attribute, it never enters the index at all. DynamoDB just skips it.

Picture an orders table with 20 million rows. Maybe 1% are still open and need processing. Put an attribute called is_open on only those, leave it off everything else, and build a GSI keyed on it. Your index holds a few thousand rows instead of 20 million. Queries run in single-digit milliseconds, storage costs almost nothing, and you wrote zero filtering logic.

When an order ships, delete the attribute. DynamoDB removes it from the index for you. The index garbage collects itself as work moves through your system.

One trap before you go build this. If you're using LSIs instead, every item that shares a partition key has to fit in 10 GB, and that includes every LSI copy. Three LSIs means four copies of each item counting against the same ceiling. Cross it and DynamoDB starts rejecting writes for that customer. You can't drop an LSI without rebuilding the whole table.

And if you're still gluing strings together like SHIPPED#us-east-1 to fake a composite key, stop. Since November 2025 a GSI can take up to four partition key attributes and four sort key attributes natively. No synthetic fields, no backfills. Sort keys match strictly left to right, so define them in the order you actually query them.

I put the whole mental model into 25 minutes, including the write sharding pattern for when adaptive capacity can't save you: https://youtu.be/xVsEviV2vNA

Day 9/30 AWS System Design Patterns

Joud Awad — Wed, 29 Jul 2026 17:46:10 +0000

A lending platform runs its loan approval workflow as a Lambda durable function (Lambda's checkpoint-and-replay execution mode — on every resume, the handler re-runs from the top and skips completed durable operations using their stored results). The flow: validate the application (a context.step()), pull the credit report (a step), then a review loop — context.wait() for 6 hours, check whether a human reviewer has approved, repeat for up to 5 days. Function timeout: 90 seconds per invocation. Execution timeout: 7 days.

At the top of the handler — before any step — the code downloads a 40 MB compliance ruleset from S3 (object storage) and parses it. It takes about 50 seconds. The engineer put it there deliberately: "several steps need it, so load it once."

The workflow ships. Applications flow through reviews for two days. Then, gradually, applications stop progressing. By Thursday, 1,900 applications are frozen mid-review. The business logic has thrown zero errors. The durable execution history shows executions alive and well within their 7-day window — just never advancing past their latest wait. The only anomaly: invocation duration on the function is pinned at exactly 90 seconds.

What is happening?

A) The executions exceeded the durable execution timeout — but the execution timeout (7 days here, up to 1 year) is a separate setting from the Lambda function timeout, and the history shows executions only 2–3 days old and still active

B) context.wait() keeps the function running and billing during the 6-hour waits, and the accumulated wait time consumed the timeout — but a wait suspends the execution entirely; each resume is a brand-new invocation with a fresh 90-second function timeout

C) The 50-second ruleset load sits outside any durable operation — so it is not checkpointed, and it re-executes on every single resume; each resume pays 50 seconds of S3 load plus replay of a growing checkpoint log before reaching any new work, and after enough review cycles a resume can no longer finish inside 90 seconds — it is killed mid-replay, retried, and killed again

D) The checkpoint log hit its per-execution size limit and Lambda silently stopped scheduling replays — but exceeding durable execution limits surfaces as explicit errors in the execution history, not a silent stall, and the 90-second duration signature points at the invocation clock

Answer in the comments.

Day 8/30 AWS System Design Patterns

Joud Awad — Tue, 28 Jul 2026 15:44:11 +0000

One malformed record. 61,000 duplicate loyalty credits.

A retail platform streams purchase events into a Kinesis Data Stream (streaming log — consumers track their own position via a checkpoint; the stream never deletes on read). A Lambda function (serverless compute) consumes the stream through an event source mapping (polls the shard and invokes Lambda with batches — advances the checkpoint only when the whole batch succeeds) with a batch size of 100. For each record, the function calls the loyalty service and credits points to the customer's account.

The pipeline has run cleanly for a year at 2 million events per day.

On Tuesday at 9:40 AM, a producer deployment ships a bug: one purchase event is published with a null user_id. It lands at position 73 of a batch. The function processes records 1 through 72 — credits 72 customers — then throws on record 73. The invocation fails.

The event source mapping retries. Not record 73 — the batch. Records 1 through 72 are credited again. Record 73 throws again. Retry. Again. The mapping's retry setting is the default: keep retrying until the record ages out of the stream's 24-hour retention.

Support notices at 1:50 PM: four hours, roughly 850 retry cycles, 61,000 duplicate credits, and every purchase event behind the bad record on that shard is stuck waiting. No Lambda alarm fired — the function's errors look like a routine blip at first glance, because the error count is one per retry, not 61,000.

Why did already-processed records run again, and what is the correct fix?

A) Kinesis (streaming log) delivered the same records multiple times during the incident — but a stream is not a delivery service that can duplicate; the consumer reads from a position it controls, and the stream served exactly what was asked for

B) The checkpoint advances per batch, not per record — a thrown error rewinds the consumer to the last committed checkpoint, so every record after it, including the 72 that succeeded, re-executes on every retry until the batch finally succeeds or the poison record expires

C) Two concurrent Lambda invocations processed the same shard and raced — but the event source mapping invokes one batch at a time per shard precisely to preserve ordering; there is no concurrent second reader on the shard

D) The loyalty service retried the credits internally — but it received separate, fully-formed requests carrying no shared idempotency key; from its side these were 850 distinct instructions to credit points, and it executed them correctly

Answer in the comments.

Day 7/30 AWS System Design Patterns

Joud Awad — Mon, 27 Jul 2026 20:40:47 +0000

A multi-tenant SaaS platform sends all background notifications — email, SMS, webhook calls — through a single SQS Standard queue (message queue — a message is removed only when a consumer explicitly deletes it or its retention period expires, 4 days by default). The queue feeds a Lambda function (serverless compute) through an event source mapping (polls the queue, invokes Lambda with batches, and deletes messages when the invocation succeeds). The team did things right: the queue has a Dead Letter Queue (DLQ — a second queue that receives messages after they fail delivery more than maxReceiveCount times) with maxReceiveCount of 5, and an alarm on DLQ depth.

Friday, 6 PM: a routine deployment ships two changes. One updates the email provider SDK — and breaks the provider credential lookup. The other is a "hardening" change from code review: the handler body is wrapped in a try/catch that logs any exception and returns normally, "so one bad notification can't poison the batch."

Saturday, 9 AM: support is flooded. No customer received anything overnight — 41,000 notifications gone. The team checks CloudWatch (AWS monitoring service): zero function errors all night. Queue depth: zero. The DLQ: empty. The DLQ alarm never fired. Every dashboard is green.

Where did 41,000 messages go?

A) The messages crossed an internal receive-count threshold and SQS deleted them — but no such mechanism exists; receive count only routes messages to a DLQ via a configured redrive policy, and SQS deletes nothing based on how many times a message was received

B) The handler caught every exception and returned success — Lambda reported a clean invocation, so the event source mapping did what success means: it called DeleteMessage on every message in the batch; the messages were deleted legitimately, one green invocation at a time

C) The queue's retention period expired overnight — but retention defaults to 4 days; messages a few hours old cannot age out, and expiry would not explain the zero-error, empty-DLQ picture either way

D) The deployment disabled the event source mapping, so messages were never consumed — but an unconsumed queue shows growing depth; a queue at zero depth with no errors means messages were received, "processed," and deleted

Answer in the comments.

Redis Cluster Mode Explained

Joud Awad — Mon, 27 Jul 2026 11:10:54 +0000

Your Redis node is out of memory.

You add three more nodes. Still full, and nothing got faster.

You added copies of your data when what you actually needed was room for it.

Replication and sharding solve completely different problems. A replica is a full copy of the primary, which is great for reads and failover, but it buys you zero extra space. Sharding splits the key space so each primary owns a different slice. One cake cut across three plates, not the same cake run through a photocopier.

Cluster mode is how Redis does the slicing: 16,384 hash slots, CRC16 of the key picks the slot, and the slot owns the key, not the node.

What nobody warns you about is that flipping it on changes your application code:

→ MGET, MULTI, and Lua only run when every key hashes to the same slot. Otherwise you get CROSSSLOT.
→ Hash tags like {42} force keys together, and overusing them rebuilds the exact bottleneck you were escaping.
→ Only DB 0 exists. SELECT is gone.
→ Your one giant leaderboard still sits on a single node, because a key cannot be split. No amount of re-sharding fixes that. It's a data model change.

I broke the whole thing down visually: slots, MOVED vs ASK, gossip and failover, the async durability gap, and when ElastiCache's toggle actually earns its keep.

https://youtu.be/0G5_w2lX02o

Day 6/30 AWS System Design Patterns

Joud Awad — Sun, 26 Jul 2026 12:16:09 +0000

A SaaS platform uses EC2 Auto Scaling (automatically launches or terminates EC2 instances based on demand) to run its API tier. Instances run the application server, hold in-memory session state for active requests, and drain gracefully on shutdown by finishing in-flight requests before terminating.

The platform deploys a new version via a rolling update. The Auto Scaling group terminates old instances and launches new ones. During each termination event, the ALB (Application Load Balancer, stops routing new requests to draining targets) deregisters the instance and marks it as draining — but active requests that were mid-flight at the moment of deregistration return 502 errors to clients.

The engineering team increases the ALB deregistration delay from 30 seconds to 120 seconds. The 502 errors continue.

What is the actual problem?

A) The ALB deregistration delay (tells ALB to wait N seconds before fully deregistering — allows in-flight requests to complete) is still too short — increase it to 300 seconds to give all in-flight requests time to complete; but the 502s continued after increasing from 30 to 120 seconds — if delay length were the only problem, the errors would have decreased, not continued unchanged

B) The EC2 Auto Scaling lifecycle hook (pauses instance termination until your code signals completion — coordinates ALB drain timing with actual EC2 termination) is not configured — without it, instances are terminated immediately after receiving the termination signal, before the ALB deregistration delay can drain connections

C) The application server is not handling SIGTERM (Linux signal sent to process on termination — process should finish in-flight work then exit cleanly) — it exits immediately on the termination signal instead of waiting for in-flight requests to finish

D) The ALB (Application Load Balancer) is sending new requests to the draining instance during the deregistration window — this is a known ALB bug at high connection counts; but ALBs do not route new requests to targets in draining state — this is not a bug

Answer in the comments.

Day 5/30 AWS System Design Patterns

Joud Awad — Sat, 25 Jul 2026 16:06:23 +0000

A checkout API runs on Lambda (serverless compute, Node.js, 1,024 MB, attached to a VPC for RDS access). Cold starts were causing P99 spikes of 3.8 seconds on Monday mornings when traffic ramps up after the weekend.

The team enabled Provisioned Concurrency (pre-initializes a fixed number of execution environments so they are warm before requests arrive — billed per environment, around the clock, whether or not traffic uses them): 10 pre-initialized execution environments, configured to stay warm at all times. Additional monthly cost: approximately $110.

The following Monday, P99 still spiked — this time to 3.1 seconds on the first wave of requests. The provisioned environments showed 0 invocations in CloudWatch (AWS monitoring service) during the spike window. The function was being invoked. The provisioned environments were running. Nothing was using them.

What went wrong?

A) 10 provisioned environments is not enough — the Monday spike requires at least 50 to absorb the concurrency burst

B) Provisioned Concurrency was configured on the $LATEST version (the mutable, always-updating function version — Provisioned Concurrency cannot target it) — it must be applied to a published version or an alias pointing to one, and the event source must invoke that version or alias

C) The VPC attachment (places Lambda inside the customer VPC for private resource access) is re-initializing the ENI on each provisioned environment at invocation time, overriding the warm state

D) Provisioned Concurrency requires the function to be behind API Gateway (managed API layer) — direct invocation bypasses the warm pool

Answer in the comments.

Day 4/30 AWS System Design Patterns

Joud Awad — Fri, 24 Jul 2026 19:56:35 +0000

A machine learning platform runs inference workloads on ECS (Elastic Container Service — managed container orchestration) with EC2 launch type. The service auto-scales based on CPU utilization. The container image is 2.8 GB — it includes a PyTorch model baked directly into the image. The image size is fixed: the model cannot be externalized.

During off-peak hours, the service runs on 4 instances. At 9 AM every weekday, a traffic spike triggers auto scaling. Eight new EC2 instances (virtual machines — pull container images from ECR before the first container can start) are launched. Each one pulls the 2.8 GB image from ECR (Elastic Container Registry — managed Docker image registry) before the first task can start.

Time from instance launch to first healthy ALB (Application Load Balancer) target: 6 minutes 40 seconds. The SLA for scale-out completion is 3 minutes. Every morning, the platform breaches it.

The team accepts that the image size cannot change. A senior engineer says there are two distinct solutions here — and they solve the problem differently depending on whether the instance has seen the image before.

Which answer correctly describes both mechanisms and when each applies?

A) Set ECSIMAGEPULL_BEHAVIOR=prefer-cached (skips ECR pull if the image is already on the instance disk) on all instances — this skips the ECR pull entirely for new instances that have never seen the image, solving the cold scale-out problem; but prefer-cached only helps if the image is already cached locally — new instances have nothing cached

B) Use SOCI indexing (Seekable OCI — lazy-loads only needed layers at startup, works on Fargate) in ECR for Fargate — Fargate detects the index and starts the container without downloading the full image; apply prefer-cached on EC2 for warm instances; but this scenario is EC2 launch type, not Fargate — and prefer-cached only helps warm instances that already have the image cached

C) Use SOCI indexing (Seekable OCI — lazy-loads only needed layers at startup; managed on Fargate, available on EC2 by running the soci-snapshotter with containerd on your instances) in ECR for the cold scale-out problem — new instances start containers without downloading the full 2.8 GB; use ECSIMAGEPULL_BEHAVIOR=prefer-cached on EC2 for warm instances to skip re-pulling an image already on disk

D) Pre-warm all auto-scaling instances by running a scheduled Lambda (serverless compute) that pulls the image to each new EC2 instance before it joins the ASG (Auto Scaling Group); this adds custom pre-warm scripts, version drift risk, and ASG lifecycle hook complexity that SOCI solves natively

Answer in the comments.

Day 3/30 AWS System Design Patterns

Joud Awad — Thu, 23 Jul 2026 14:36:01 +0000

340 customers were charged twice. No one touched the code.

An e-commerce platform processes orders through a simple pipeline: EventBridge (serverless event bus, invokes Lambda asynchronously — Lambda retries up to 2 times on any thrown error) receives an order.placed event, routes it to a Lambda (serverless compute), which charges the payment and writes the confirmed order to DynamoDB (managed NoSQL database). The architecture ran in production for 4 months — 8,000 orders per day, zero duplicate charges, zero incidents.

The platform runs a promotional campaign. Traffic spikes 40× over 2 hours. 14,000 orders in one window.

Post-campaign, the payments team pulls the report. 340 orders show two charges. CloudWatch (AWS monitoring service) shows EventBridge delivered each event exactly once. The Lambda has no retry logic. No alerts fired during the campaign.

The engineer on-call checks the Lambda logs. Every invocation shows a successful payment charge. Then — on 340 of them — a DynamoDB write error at the very end. Then a second invocation of the same function, for the same order, a few seconds later.

The second invocation charges the card again.

No code changed. No bug was introduced. The system behaved exactly as designed.

What is the root cause?

A) EventBridge (serverless event bus, at-least-once delivery) delivered duplicate events during the spike — at-least-once delivery means duplicates increase under high throughput

B) Lambda (serverless compute, retries async invocations up to 2 times on any thrown error) retried failed invocations after the DynamoDB write threw — the payment charge ran again on retry because the function is not idempotent

C) DynamoDB (managed NoSQL database, can throttle writes under sudden traffic spikes) throttling caused a race condition between two concurrent Lambda invocations processing the same event

D) The payment provider has its own retry logic — the platform's architecture is correct, the provider caused the double charge

Answer in the comments.