DEV Community: Majd-sufyan

Async & Messaging: System Design Journey... Week 6

Majd-sufyan — Mon, 21 Sep 2026 15:16:15 +0000

Week 6: Async & Messaging: Designing Systems That Don't Have to Do Everything Now

Overview

In Week 6, I focused on a part of system design that changes how I think about communication between components:

Synchronous vs asynchronous processing
Queues and message brokers
Pub/Sub and event-driven architecture
Message delivery guarantees
Retries and Dead Letter Queues
Idempotency and duplicate messages
Ordering and failure handling

The main idea I took away this week:

Not every piece of work needs to happen while the user is waiting for the response.

Moving work to asynchronous processing can improve responsiveness, absorb traffic spikes, and reduce coupling between services, but it also introduces a completely new set of failure scenarios that we need to design for.

Synchronous vs Asynchronous Processing

Let's start with something simple.

Imagine an e-commerce system where a user creates an order.

A synchronous approach could look like this:

User
  ↓
Order Service
  ↓
Payment
  ↓
Inventory
  ↓
Email
  ↓
Response

Everything happens before the user receives a response.

This is straightforward, but it creates a problem:

If one of those operations is slow or unavailable, the entire request can become slow or fail.

With asynchronous processing, we can separate the work:

User
  ↓
Order Service
  ↓
Queue
  ↓
Response

Then:

Queue
  ↓
Order Worker
  ↓
Payment
  ↓
Inventory
  ↓
Email

The user doesn't need to wait for every operation to finish.

The system can acknowledge the request and process the remaining work in the background.

Why Use Asynchronous Processing?

There are several reasons.

1. Reduce user-facing latency

Suppose sending an email takes a couple of seconds.

There is usually little value in forcing the user to wait for the email service before receiving an order confirmation.

Instead:

Create order
     ↓
Queue email task
     ↓
Respond to user

The email can be processed separately.

2. Handle traffic spikes

Imagine a system normally receives:

100 orders/minute

but suddenly receives:

10,000 orders/minute

A queue can act as a buffer:

10,000 requests
       ↓
     Queue
       ↓
Workers process at manageable speed

Instead of forcing every downstream service to immediately process the entire spike.

3. Decouple services

Without messaging:

Order Service
      ↓
Payment Service
      ↓
Inventory Service
      ↓
Email Service

The Order Service becomes tightly connected to everything it needs to call.

With messaging:

             ┌→ Payment
             │
Order → Queue├→ Inventory
             │
             └→ Email

Different consumers can process the work independently.

This creates a more loosely coupled architecture.

Producer and Consumer

Two terms appear constantly when working with messaging systems:

Producer

The component that creates or sends a message.

Order Service
      ↓
   Producer
      ↓
    Queue

Consumer

The component that reads and processes the message.

Queue
  ↓
Consumer
  ↓
Payment Service

The basic pattern is:

Producer → Queue → Consumer

The message might represent work that needs to happen:

{
  "type": "PaymentRequested",
  "orderId": "12345"
}

Or an event that something already happened:

{
  "type": "OrderCreated",
  "orderId": "12345"
}

Queues vs Pub/Sub

One distinction I wanted to understand clearly this week was the difference between a queue and Pub/Sub.

Queue

A queue is generally about work that needs to be processed.

For example:

             ┌→ Worker 1
Queue ───────┼→ Worker 2
             └→ Worker 3

Multiple workers can share the workload.

The queue allows the system to buffer work and process it as capacity becomes available.

A simple way to think about it:

"Someone needs to do this work."

Pub/Sub

Pub/Sub is more about broadcasting an event to multiple interested consumers.

For example:

                 ┌→ Email
                 │
OrderCreated ────┼→ Analytics
                 │
                 └→ Inventory

The Order Service doesn't necessarily need to directly call each service.

It publishes:

OrderCreated

and different consumers can react to that event independently.

The mental model I use is:

Queue = process this work.

Pub/Sub = something happened; whoever cares can react.

Event-Driven Architecture

This leads to a bigger concept: event-driven architecture.

Instead of building a chain of direct service calls:

Order Service
     ↓
Inventory Service
     ↓
Email Service
     ↓
Analytics Service

we can have:

Order Service
     ↓
OrderCreated
     ↓
Message Broker
     ↓
 ┌───┼────┬──────┐
 ↓   ↓    ↓      ↓
Email Inventory Analytics Fraud

The Order Service is essentially saying:

"An order was created."

It doesn't necessarily need to know exactly which services are listening.

This can make systems more loosely coupled and easier to extend.

For example, adding a new analytics consumer doesn't necessarily require changing the Order Service itself.

The Problem With Asynchronous Systems

This is where the topic becomes much more interesting.

Once we introduce queues, we also introduce new failure modes.

Imagine:

Queue
  ↓
Order Worker
  ↓
Charge Customer
  ↓
Update Database

What happens if the worker crashes after charging the customer but before acknowledging the message?

Queue
  ↓
Order Worker
  ↓
Charge Customer
  ↓
💥 CRASH

The system may not know whether the message was successfully processed.

Should it try again?

This leads to message delivery guarantees.

At-Most-Once Delivery

At-most-once means:

A message is processed zero or one time.

The message might be lost if processing fails.

Message
   ↓
Consumer
   ↓
Failure

No retry means no duplicate processing, but the message may never be processed.

This can be acceptable for some non-critical use cases, such as certain analytics events.

At-Least-Once Delivery

At-least-once means:

The system tries to ensure the message is processed, but it may be processed more than once.

For example:

Message #123
     ↓
Consumer
     ↓
Process message
     ↓
💥 Crash before acknowledgement
     ↓
Message delivered again

Now:

Message #123
     ↓
Processed
     ↓
Processed AGAIN

This is one of the most important concepts from this week.

At-least-once delivery means consumers need to be prepared for duplicates.

Exactly-Once Delivery

Exactly-once sounds like the ideal solution:

Process every message exactly once.

In distributed systems, however, guaranteeing true exactly-once processing across multiple components can be difficult.

For example:

Consumer
   ↓
Payment Provider
   ↓
Database

The payment might succeed, but the consumer could crash before recording that success.

When the message is retried, the consumer may not know whether the original payment happened.

Because of this, a common design approach is:

At-least-once delivery + idempotent consumers.

Rather than assuming duplicates will never happen, we design the system so duplicates are safe.

Idempotency

This was one of the concepts from Week 4 that became much more concrete this week.

An operation is idempotent when performing it multiple times produces the same final result as performing it once.

For example:

SET status = "ACTIVE"

Doing this once or ten times still results in:

ACTIVE

But:

balance = balance + €100

is not naturally idempotent.

Doing it twice gives:

+€200

instead of:

+€100

This becomes particularly important for payments and other operations where duplicate processing has real consequences.

Handling Duplicate Messages

One common approach is to give every event a unique ID:

{
  "eventId": "abc-123",
  "type": "PaymentRequested",
  "orderId": "order-456"
}

The consumer can keep track of processed event IDs.

If it receives:

abc-123

again, it can recognize that the event was already processed and avoid performing the operation twice.

The important principle is:

If your messaging system can deliver a message more than once, your consumer needs to be designed accordingly.

Retries

Not every failure is permanent.

A downstream service might simply be temporarily unavailable.

For example:

Attempt 1 → failure
       ↓
wait
       ↓
Attempt 2 → failure
       ↓
wait
       ↓
Attempt 3 → success

Instead of retrying immediately and indefinitely, systems commonly use:

Limited retry attempts
Backoff
Jitter

For example:

1 sec
  ↓
2 sec
  ↓
4 sec
  ↓
8 sec

This reduces the risk of making an already struggling service even more overloaded.

Dead Letter Queues

But what happens when a message keeps failing?

We don't want:

Message
  ↓
Retry
  ↓
Retry
  ↓
Retry
  ↓
Retry
  ↓
Forever...

Instead, after a certain number of failures, we can move it to a Dead Letter Queue (DLQ):

Message
   ↓
Attempt 1 ❌
   ↓
Attempt 2 ❌
   ↓
Attempt 3 ❌
   ↓
DLQ

The DLQ gives engineers a place to inspect problematic messages without allowing them to continuously interfere with normal processing.

For example, a message might end up there because of:

Invalid data
A persistent downstream failure
A software bug
An unexpected edge case

Message Ordering

Another question I considered was:

Does the order of messages matter?

Imagine these events:

1. OrderCreated
2. OrderCancelled

If they are processed in the opposite order:

OrderCancelled
      ↓
OrderCreated

the system could end up in an incorrect state.

But not every system needs strict ordering.

For example:

ProductViewed
ProductViewed
ProductViewed

may not require a specific order.

This means ordering should be treated as a business requirement, not something we automatically enable everywhere.

There is also a trade-off: stronger ordering requirements can restrict how much work can be processed in parallel.

Designing an Order Processing System

For this week's design exercise, I used an order processing system to bring these concepts together.

A simplified architecture looks like:

                    User
                      ↓
                POST /orders
                      ↓
                Order Service
                      ↓
                  Order DB
                      ↓
                OrderCreated
                      ↓
                    Queue
                      ↓
                 Order Worker
                  /    |    \
                 ↓     ↓     ↓
             Payment Inventory Email

The synchronous boundary could be:

User
 ↓
Create Order
 ↓
Save Order
 ↓
Publish message
 ↓
Return response

Then the remaining processing happens asynchronously.

The exact boundary depends on the requirements.

For example, if the user needs an immediate payment result, payment might need to remain synchronous. If the result can be processed later, it may be a good candidate for asynchronous processing.

The important design question is:

Which operations actually need to happen before the user receives a response?

Backend Lens

This week's design exercise made me look at asynchronous systems differently.

Instead of only asking:

"How do I process this message?"

I started asking:

What if the message is processed twice?

Can the consumer safely handle it?

What if processing fails halfway?

Could the operation leave the system in an inconsistent state?

What if the downstream service is temporarily unavailable?

Should we retry?

How many times?

What if the message never succeeds?

Should it go to a DLQ?

Does ordering matter?

If so, what level of ordering is actually required?

These questions are where a simple queue turns into a real system-design problem.

Key Takeaways

The biggest lessons I took from Week 6 were:

1. Async processing isn't just about performance

It can also help with:

Decoupling
Traffic spikes
Resilience
Independent scaling

2. Async systems introduce new failure modes

Once work becomes asynchronous, we have to think about:

Duplicates
Lost messages
Retries
Partial failures
Ordering

3. At-least-once delivery changes how consumers are designed

If duplicates are possible, consumers need to be able to handle them safely.

4. Idempotency is extremely important

Especially for operations such as payments, where processing the same event twice can have real consequences.

5. Don't make everything asynchronous

The important question isn't:

"Should this be async?"

It's:

"What actually needs to happen before I can respond to the user?"

Reflection

Week 6 helped me connect several concepts from the previous weeks.

In Week 4, I learned about retries, timeouts, circuit breakers, and idempotency in the context of distributed calls.

This week, I saw the same ideas from a different perspective.

When communication becomes asynchronous, failure is no longer just a request that returns an error.

A message can be delayed, duplicated, processed partially, retried, or moved to a dead letter queue.

That means reliable asynchronous systems aren't simply about adding a queue.

They're about designing what happens when things don't go as planned.

And that is probably the biggest lesson I took from this week:

Asynchronous architecture gives you more flexibility, but it also makes failure handling part of the design itself.

What's Next?

Next week, I'll dive deeper into distributed systems and consistency, including replication, consistency models, and the trade-offs involved when data exists across multiple machines.

APIs & Service Architecture. System Design Journey... Week 5

Majd-sufyan — Tue, 08 Sep 2026 14:01:14 +0000

Learn System Design with Me — Week 5 / 12

APIs & Service Architecture

Overview

In Week 5, I shifted my focus from how systems store and process data to how different clients and services communicate with them.

The main goals this week were to:

Understand the differences between REST, GraphQL, and gRPC
Learn how to think about API design as part of system architecture
Understand API versioning and why it matters
Design an API that can support multiple clients, such as web and mobile
Redesign an existing API with scalability and maintainability in mind

One thing that became clear this week is that an API is much more than a collection of endpoints.

It is a contract between systems.

REST vs GraphQL vs gRPC

One of the main topics this week was understanding that there is no universally "best" API technology.

The right choice depends on the communication pattern and requirements of the system.

REST

REST is probably the style I've worked with the most throughout my career.

It is resource-oriented and commonly uses HTTP methods such as:

GET
POST
PUT
PATCH
DELETE

For example:

GET /users/123

REST is simple, widely supported, and works very well for public-facing APIs.

GraphQL

GraphQL takes a different approach.

Instead of the server defining exactly which fields are returned by each endpoint, the client can request the data it needs.

For example, a mobile application might only need:

{
  user {
    id
    name
    avatar
  }
}

while a web application might request additional information.

This can be especially useful when different clients have very different data requirements.

However, GraphQL also introduces additional complexity around:

Query complexity
Caching
Authorization
Monitoring
Preventing expensive queries

gRPC

gRPC is designed more for service-to-service communication than typical public APIs.

It uses Protocol Buffers and provides strongly typed contracts between services.

For example:

Order Service
      ↓
   gRPC call
      ↓
Payment Service

The main takeaway for me was:

REST, GraphQL, and gRPC aren't competitors where one simply replaces the others.

They solve different communication problems.

A system can even use multiple approaches at the same time.

For example:

Web / Mobile
     ↓
  REST API
     ↓
Backend Services
     ↓
    gRPC
     ↓
Other Internal Services

API Versioning

Another important topic this week was API evolution.

APIs often live much longer than the code that initially created them.

Once clients depend on an API, changing its contract can break those clients.

For example, imagine we initially have:

GET /api/v1/users/123

and the response contains:

{
  "id": 123,
  "name": "John"
}

Later, we want to make a breaking change to the response.

Instead of immediately breaking existing clients, we can introduce:

GET /api/v2/users/123

This allows older clients to continue using v1 while newer clients migrate to v2.

The important lesson wasn't simply "use /v1 and /v2."

It was understanding that API contracts need to evolve without unnecessarily breaking consumers.

Designing a Multi-Client API

For this week's design exercise, I considered a system used by both:

Web clients
Mobile clients

At first this sounds simple.

But the requirements of the clients can be very different.

A web application might have:

Large screen
Fast network
More bandwidth
More complex UI

A mobile application might have:

Limited bandwidth
Higher latency
Smaller payload requirements
Unreliable network connections

This creates an important API design question:

Should every client receive exactly the same data?

Not necessarily.

API Design Trade-offs

One approach is to create a single REST API that serves all clients.

              ┌── Web
              │
Client ───────┼── Mobile
              │
              └── Other clients
                    ↓
                 REST API

This is simple and easy to maintain initially.

But as clients become more complex, the API can start returning either:

Too much data
Too little data

This is one reason technologies such as GraphQL can become attractive for multi-client systems.

The important part is not choosing GraphQL automatically.

It is recognizing the trade-off between simplicity and flexibility.

API as a Contract

One of my biggest takeaways from this week was thinking about APIs as contracts.

When another system consumes your API, it depends on things such as:

Request format
Response structure
Error behavior
Authentication
Performance expectations
Versioning

Changing an API therefore isn't just a code change.

It can become a system-wide change.

This is why API design should consider not only how an endpoint works today, but also how it might evolve in the future.

Redesigning an Existing API

As part of the practical work, I also looked at how an existing API could be redesigned.

Instead of only asking:

"Does this endpoint work?"

I started asking:

Who consumes this API?
What data does each client actually need?
Which operations are read-heavy?
What happens when the API evolves?
Which changes would break existing clients?
Should this communication use REST, GraphQL, or gRPC?
How should errors be represented?
How should the API be versioned?

These questions make API design feel much more like a system design problem.

Reflections

Before this week, I mostly thought about APIs from the perspective of implementing endpoints.

This week made me think about APIs from the perspective of communication between systems.

The technology is only one part of the decision.

The more important questions are:

Who is consuming the API?

What are their requirements?

How will the API evolve?

What happens when there are hundreds of clients depending on it?

That shift in perspective was probably the biggest lesson from Week 5.

What’s Next — Week 6

Next week, I'll move deeper into distributed systems and asynchronous communication:

Message queues
Event-driven architecture
Kafka
Asynchronous processing
Designing systems that don't need every operation to happen synchronously

The journey continues 🚀

Reliability, Failures & Designing a Payment API: System Design Journey... Week 4

Majd-sufyan — Thu, 25 Jun 2026 18:46:00 +0000

Overview

In Week 4, I focused on a topic that every distributed system eventually faces:

Failures are inevitable.

No matter how well a system is designed, networks fail, servers crash, databases become unavailable, and requests time out.

The goal this week was to understand how reliable systems continue operating despite these failures.

My main focus areas were:

Fault tolerance
Retries and timeouts
Circuit breakers
Idempotency
Designing systems that avoid cascading failures

To apply these concepts, I designed a simplified Payment API, where correctness matters more than almost any other requirement.

Reliability vs Availability

One idea that stood out immediately is that a system can be available without being reliable.

For example:

A payment service might always respond
But accidentally charge a customer twice

Technically, the system is available.

But it is not reliable.

This changed how I think about backend systems.

Users care less about whether a request returns a response and more about whether the system behaves correctly.

Timeouts, Retries & Circuit Breakers

Most distributed systems communicate over unreliable networks.

Sometimes a request succeeds, but the response never arrives.

Sometimes a downstream service becomes slow.

Sometimes it becomes completely unavailable.

Timeouts

A timeout prevents requests from waiting forever.

Instead of hanging indefinitely, the request fails after a predefined period.

This protects resources and prevents thread exhaustion.

Retries

Retries allow temporary failures to recover automatically.

Examples:

Temporary network issue
Short database outage
Service restart

However, retries can also be dangerous.

If thousands of clients immediately retry a failing service, they can amplify the outage.

This is known as a retry storm.

Circuit Breakers

Circuit breakers help prevent cascading failures.

When a downstream service starts failing repeatedly:

New requests are stopped early
The service is given time to recover
Resources are protected

A circuit breaker acts similarly to an electrical fuse.

Instead of allowing one failure to spread across the system, it isolates the problem.

Idempotency: The Most Important Concept This Week

The biggest lesson from Week 4 was idempotency.

An operation is idempotent when performing it multiple times produces the same result as performing it once.

For example:

Creating a payment is not naturally idempotent.

If a payment request is processed twice:

The customer may be charged twice
Financial records become inconsistent
Customer trust is lost

To solve this problem, payment APIs typically require an Idempotency Key.

The client sends a unique identifier with the request:

POST /payments

Idempotency-Key: abc123

If the same request is retried:

The server recognizes the key
Returns the original result
Prevents duplicate charges

This allows clients to safely retry requests when failures occur.

Applying the Concepts: Designing a Payment API

To practice these ideas, I designed a simplified payment processing system.

Functional Requirements

Create payments
Retrieve payment status
Prevent duplicate charges
Return payment history

Non-Functional Requirements

High reliability
Strong consistency
Low latency
Fault tolerance
High availability

Unlike previous systems, correctness is more important than raw performance.

High-Level Architecture

The system consists of:

Stateless API servers
PostgreSQL for durable storage
Redis for idempotency lookups
Load balancer
Payment processor integration

Request flow:

Client submits payment request
API validates the idempotency key
Payment is stored in the database
The external payment provider is called
The result is returned to the client

Failure Scenarios

One of the most useful exercises this week was identifying failure modes.

This exercise reinforced an important lesson:

Designing for failure is often more important than designing for success.

What Changed in My Thinking

Before this week, I often thought about performance first.

Now I find myself asking different questions:

What happens if this request runs twice?
What happens if the downstream service fails?
What happens if the response never arrives?
What happens if retries overload the system?

These questions feel much closer to how real production systems are designed.

Reflections

Week 4 was less about scaling and more about correctness.

The most valuable takeaway was realizing that distributed systems spend a surprising amount of time handling situations where things go wrong.

Reliability is not achieved by preventing failures.

It is achieved by expecting failures and designing systems that can recover from them.

What’s Next — Week 5

Replication
Consistency models
Read replicas
Leader-follower architectures

The journey continues 🚀

Caching: System Design Journey... Week 3

Majd-sufyan — Wed, 20 May 2026 15:09:12 +0000

In Week 3, I focused on one of the most important concepts in scalable backend systems:

Avoid hitting the database unnecessarily.

After learning how databases store and retrieve data in Week 2, the next logical question became:

What happens when traffic grows?
What if millions of users request the same data repeatedly?
How do systems remain fast under heavy read load?

This week was all about caching.

The main goals were to:

Understand why caching exists
Learn common cache strategies
Explore cache invalidation trade-offs
Apply these concepts by designing a high-traffic product catalog system

Compared to previous weeks, this felt like the point where system design started becoming much closer to real-world backend engineering.

*Why Caching Exists *
Databases are powerful, but they are also expensive resources.

Even optimized relational databases eventually become bottlenecks under massive read traffic.

Imagine a product page on an e-commerce platform:

A product may be viewed millions of times
But the actual product data changes very rarely

Without caching:
Client → API → Database
for every request.

With caching:
Client → API → Cache → Response
The database is queried once, and the cache serves the remaining requests.

This dramatically reduces:

Latency
Database load
Infrastructure costs

Key Insight: Caching Improves Performance but Adds Complexity

One of the biggest lessons this week was:

Cache is easy to add, but difficult to keep correct.

Caching introduces a new challenge:

What happens when cached data becomes outdated?

This is the famous problem of cache invalidation.

Cache Invalidation: The Hard Problem

Example:

A product is cached with:

Price = $999

An admin later updates the database:

Price = $899

But the cache still returns:

$999

This creates stale data.

The core challenge becomes:

How do we keep cache and database synchronized without hurting performance?

Cache-Aside Pattern

The main strategy I focused on was the cache-aside pattern.

Flow:

1. Request arrives
2. Check cache

IF HIT:
    return cached data

IF MISS:
    query database
    store result in cache
    return response

This approach is:

Simple
Common in production systems
Easy to implement with Redis

It also reinforced another important principle:

Cache should be treated as an optimization layer, not the source of truth.

Applying the Concepts: High-Traffic Product Catalog

To apply these ideas, I designed a simplified high-traffic product catalog system similar to what platforms like Amazon or Shopify might use.

Functional Requirements

View products
Search products
Update product details
View featured / trending products

Non-Functional Requirements

Very low latency
High read throughput
High availability
Scalability under heavy traffic
Fault tolerance

High-Level Architecture

Figure 1: Product catalog architecture with Redis caching layer.

The system consists of:

Stateless API servers behind a load balancer
PostgreSQL as the primary data store
Redis as the caching layer
Cache-aside strategy for reads

The majority of requests are served directly from Redis, dramatically reducing pressure on the database.

What Should Be Cached?

This week also helped me think more carefully about what should actually be cached.

Good candidates for caching:

Product details
Product lists
Trending products
Frequently viewed items

Less ideal candidates:

Rapidly changing inventory
Real-time stock counts
Frequently updated pricing

This highlighted an important design trade-off:

The more dynamic the data, the harder caching becomes.

TTL (Time To Live)

Another concept I explored was TTL (Time To Live).

Instead of keeping cached data forever, cached entries automatically expire after a period of time.

Example:

Product cache expires after 5 minutes
Benefits:

Simpler invalidation strategy
Prevents very stale data
Reduces cache management complexity

Trade-off:

Slightly outdated data may temporarily exist

Failure Considerations

The system was designed so that:

If Redis fails → the database still serves requests
If API instances crash → traffic is routed elsewhere
If traffic spikes → stateless services scale horizontally

This reinforced a recurring system design principle:

Reliability often comes from graceful degradation.

Even if performance drops, the system should continue functioning correctly.

Implementation Notes

The product catalog system was implemented using:

Kotlin + Spring Boot
PostgreSQL
Redis
Cache-aside caching strategy

The focus this week was less about framework syntax and more about understanding:

access patterns
cache trade-offs
latency optimization
Key Takeaways

This week helped me understand that:

Databases alone are not enough at scale
Caching is fundamental for read-heavy systems
Cache invalidation is where complexity begins
Not all data should be cached equally
System design is mostly about trade-offs

What’s Next — Week 4

Replication & consistency
Primary databases vs read replicas
Designing systems that survive failures and replication lag The journey continues 🚀

Storage & Retrieval: System Design Journey... Week 2

Majd-sufyan — Thu, 09 Apr 2026 11:07:33 +0000

In Week 2, I shifted focus from high-level scalability concepts to how data is actually stored, retrieved, and optimized in real systems.

The main goals this week were to:

Build intuition around storage engines and access patterns
Understand SQL vs NoSQL trade-offs
Learn where indexes and caching fit into system design
Apply these ideas by designing and implementing a User Profile Service

Compared to Week 1, this week went deeper into backend fundamentals—and required slowing down to truly understand what happens under the hood.

Key Learnings from Designing Data-Intensive Applications — Chapter 3

Chapter 3 (Storage & Retrieval) was significantly more challenging than earlier chapters—but also much more rewarding.

Storage Engines: Why Data Layout Matters

One of the biggest takeaways was that databases are not magic. They rely on concrete data structures that directly impact performance.

Two broad approaches stood out:

Log-structured storage
- Writes are fast (append-only)
- Reads require compaction and indexes
B-tree–based storage
- Optimized for reads
- Supports range queries efficiently
- Common in relational databases (e.g., PostgreSQL)

This helped me understand why certain databases excel at specific workloads.

The way data is written on disk determines what queries are fast—or painfully slow.

Indexes: Trading Space for Speed

Indexes are one of the most important tools for performance, but they are not free.

Key insights:

An index is a separate data structure (not just “extra metadata”)
Indexes speed up reads but:
- Slow down writes
- Increase storage usage
Indexes should reflect real query patterns, not hypothetical ones

This clarified a common misconception I had earlier:

Indexes don’t automatically exist for every field—you must design them intentionally.

SQL vs NoSQL: It’s About Trade-offs, Not Trends

Instead of “SQL vs NoSQL”, the chapter reframed the question as:

What guarantees does your system need?

Key comparisons:

Aspect	SQL	NoSQL
Schema	Strong, enforced	Flexible
Transactions	Strong consistency	Often eventual
Scaling	Harder (sharding)	Easier horizontally
Querying	Powerful	Limited / specialized

This made it clear why relational databases are still a strong default choice, especially for systems that require correctness and structured data.

Applying the Concepts: User Profile Service

To apply these ideas, I designed and implemented a User Profile Service—a common real-world backend component.

Functional Requirements

Create user profiles
Retrieve user information
Update user details
Delete users

Non-Functional Requirements

Low latency (read-heavy system)
High availability
Scalability (growing user base)
Fault tolerance
Optional analytics

Scale Assumptions

To guide design decisions, I assumed:

Users created per day: ~10,000
Reads per day: ~1,000,000
Read/write ratio: ~100:1

This clearly indicates a read-heavy system, making caching and indexing critical.

High-Level Architecture

Figure 1: High-level flows for create, read, update, delete, and analytics.

The system consists of:

Stateless API servers behind a load balancer
PostgreSQL as the primary data store
Redis for caching user profiles
Optional analytics endpoints

Reads are optimized through caching, while writes remain strongly consistent.

Data Modeling & Indexing

The core data model includes:

userId (primary key)
User profile fields (name, address, metadata)
Timestamps for creation and updates

Where Indexes Fit

The primary key index enables fast lookup by user ID
Additional indexes are unnecessary unless new access patterns emerge (e.g., search by email)

This aligns with the lesson from DDIA:

Indexes should be driven by real queries—not assumptions.

Caching Strategy

A key design decision was what to cache.

Chosen Approach: Cache the Entire User Object

Cache key: userId
Cache value: full user profile

Why this works well:

User profiles are relatively small
Read requests usually need the full object
Simplifies cache logic

Cache Invalidation

On update → invalidate cache
On delete → invalidate cache
Cache misses fall back to the database

This follows the cache-aside pattern, which is simple and robust.

Failure Considerations

The system is designed so that:

If Redis fails → database still serves reads (slower, but correct)
If an API instance crashes → load balancer routes traffic elsewhere
If traffic spikes → stateless services scale horizontally

This reinforced an important lesson:

Cache is an optimization, not a dependency.

Implementation Notes

The service was implemented using:

Kotlin + Spring Boot
PostgreSQL
Redis
Stateless REST APIs

The focus was not on framework features, but on mapping system design decisions into real code.

What’s Next — Week 3

Replication & consistency
Read replicas vs primary databases
Designing systems that survive partial failures

The journey continues 🚀

Scalability & URL Shortener: System Design Journey... Week 1

Majd-sufyan — Thu, 26 Mar 2026 13:13:48 +0000

Over the next 12 weeks, I’m focusing on one goal:

Becoming more confident in system design by going beyond theory and applying it in practice.

I’ll be documenting everything I learn — the concepts, the mistakes, and the design decisions.

Each week focuses on a different topic, with a real system to apply it.

Week 1 — Scalability Basics & Designing a URL Shortener

Overview

In Week 1 of my system design learning plan, I focused on foundational scalability concepts and applied them by designing and implementing a URL shortener system.

The goal was not just to design a working system, but to:

Build intuition for data-intensive systems
Practice thinking in trade-offs
Connect

Key Learnings from Designing Data-Intensive Applications — Chapter 1

Data-Intensive vs Compute-Intensive Systems

Most modern applications are data-intensive, meaning the primary challenges are:

Volume of data
Speed of change
Access patterns (reads vs writes)

CPU is rarely the bottleneck.

This shifted my thinking from “what framework should I use?” to:

What data do I store?
How often is it accessed?
What guarantees does it need?

This mindset directly influenced how I designed the URL shortener.

Reliability, Scalability, and Maintainability

Chapter 1 emphasizes that failures are inevitable, but systems should be designed so faults don’t become failures.

Key takeaways:

Fault tolerance matters even for “simple” systems
Scalability is about how systems behave as load increases
Maintainability is critical because most software costs are long-term

These principles guided decisions like stateless services, caching, and asynchronous processing.

Load Balancing Fundamentals

A load balancer solves two core problems:

Availability — no single server failure brings the system down
Scalability — traffic can be distributed across multiple instances

Important insights:

Load balancers require stateless application servers
Distribution strategies (round-robin, least connections) affect latency
Health checks are essential to avoid routing traffic to failed instances

URL Shortener — System Design

Functional Requirements

Accept long URLs
Generate unique short keys
Redirect users to the original URL
Store mappings reliably

Non-Functional Requirements

Low latency (read-heavy workload)
High availability
Scalability
Fault tolerance
Optional analytics

High-Level Architecture

Figure 1: High-level request flow for URL creation, redirection, and analytics.

The system consists of:

Stateless API servers behind a load balancer
PostgreSQL for persistent storage
Redis for low-latency reads
Asynchronous analytics processing

Redirects are optimized for speed, while analytics are handled in the background.

Design Improvement: Asynchronous Analytics

Initial Approach (Synchronous)

Initially, redirect analytics were updated synchronously inside the redirect request:

val mapping = repository.findByShortUrl(shortUrl)
mapping.redirectCount +=1
repository.save(mapping)
return RedirectToOriginalResponse(mapping.originalUrl)

Problem:

The user waits for a database write that provides no immediate value.

Timeline:

UserRequest
→DBRead(10ms)
→DBWrite(10ms)
→Response

Total ≈20ms

Improved Approach (Asynchronous)

After revisiting DDIA’s discussion on latency and performance, I moved analytics updates to run asynchronously.

Now:

Redirect happens immediately
Analytics are processed in the background
Redirects remain fast even if analytics fail

Timeline:

User Request
→ DBRead (10ms)
→ Async event / queue
→ Response

User latency ≈10ms

Why This Matters

This change:

Reduced redirect latency by ~50%
Improved resilience
Accepted eventual consistency where strict accuracy isn’t required

A key system design principle reinforced here:

Not all data needs the same consistency guarantees.

Implementation

The system was implemented using:

Kotlin + Spring Boot
PostgreSQL for persistence
Redis for caching
Asynchronous analytics processing

📌 Source code:

👉 https://github.com/Majd-sufian/12-week-System-Design-Learning-Plan/tree/main/url-shorten-project

Reflections

This week helped me:

Shift from component-level thinking to system behavior thinking
Make explicit design trade-offs
Connect theory (DDIA) to real implementation decisions

Designing even a “simple” system like a URL shortener revealed how quickly concerns like latency, fault tolerance, and scalability appear.

What’s Next — Week 2

Deeper dive into databases
Data modeling & indexing
Designing read-heavy systems under higher load

The Easiest Way to Get OpenAI Keys for Free

Majd-sufyan — Fri, 30 Aug 2024 12:31:00 +0000

Ever wondered how some people get their hands on OpenAI API keys without breaking a sweat? Well, a quick look at GitHub might give you the answer! 😅

Check out this image—someone accidentally pushed their OpenAI API key right into their GitHub repo. It’s a common mistake, but it means anyone who finds it can grab that key and use it for free. Yep, just like that.

But seriously, don’t do this! To avoid leaking your own keys:

Use environment variables instead of hardcoding them.
Add .env files to your .gitignore so they don’t get uploaded.
Scan your repos for secrets with automated tools.
Rotate your keys if they ever get exposed.

Stay safe out there, and keep those keys hidden! 🔒

Explore My Portfolio and Share Your Feedback! 🕺

Majd-sufyan — Tue, 27 Aug 2024 21:24:22 +0000

I'm excited to share my new portfolio, a dynamic and interactive space where you can explore my latest projects, skills, and experiences. I would love for you to visit my portfolio, try it out, and share your feedback with me. Your input is invaluable as I continue to refine and improve my work!

Here’s a peek at the tools and frameworks that power my site:

Next.js
React
Three.js & @react-three/fiber
Framer Motion
Tailwind CSS & Tailwind Merge
Sentry

You can check out the code behind this portfolio on GitHub and explore it live at https://www.majd-sufyan.site/.

Let's Connect!

If you like what you see, I’d love to connect with you on LinkedIn. Following me there is the best way to stay updated on my latest projects and connect over shared interests.

Your feedback is crucial to me, so please don’t hesitate to reach out after exploring my portfolio. I’m looking forward to hearing your thoughts!

Good vs. great Programmer 💪

Majd-sufyan — Tue, 27 Aug 2024 21:03:58 +0000

How to Answer Technical Questions Like a Pro

Answering technical interview questions is all about showing off your problem-solving skills and understanding! A great response not only highlights what you know but also how you think. In this article, we’ll explore fun ways to improve your answers to common web and software development questions, turning good responses into fantastic ones. Stick around for some friendly tips to help you shine in your technical interviews!

Example 1: Handling a Slow Database Query

Interviewer: "What would you do if a database query is running much slower than expected in a production environment?"

Good Developer:
"I would start by checking the database indexes to ensure they are set up correctly. If needed, I would add indexes on the columns used in the query’s WHERE clause. I would also look into optimizing the query by reducing the number of joins or breaking it down into smaller queries."

Great Developer:
"First, I’d analyze the slow query using the database's query analyzer or execution plan to pinpoint where the bottleneck is. Common issues might include missing indexes, inefficient joins, or excessive data retrieval. After identifying the problem, I would optimize the query by adding or adjusting indexes, refactoring the query for better performance, and possibly denormalizing some data if appropriate. I would also consider caching frequent queries, especially if the underlying data doesn’t change often. Monitoring tools should be implemented to continuously assess query performance. If performance issues persist, I might evaluate the database structure and consider more scalable solutions like partitioning, sharding, or even migrating to a NoSQL database for specific use cases."

Example 2: Debugging a Memory Leak in a Web Application

Interviewer: "How would you approach diagnosing and fixing a memory leak in a web application?"

Good Developer:
"I would use a memory profiler to track down the memory leak. Once identified, I’d look for places in the code where objects are not being released properly, like event listeners or large data structures, and make sure they are cleaned up."

Great Developer:
"I’d begin by identifying the symptoms of the memory leak, such as increased memory usage over time or performance degradation. Using a memory profiler, I would track the application's memory usage to locate the exact spot where the leak occurs. Common culprits might include lingering event listeners, unoptimized DOM manipulation, or large data objects that aren't being properly garbage collected. After finding the issue, I’d refactor the code to ensure proper cleanup, such as removing event listeners when they are no longer needed, avoiding global variables, and optimizing data structures. Post-fix, I’d implement automated memory monitoring to catch any future leaks early. Additionally, I’d review the entire codebase for similar patterns that could cause memory issues."

Example 3: Optimizing Front-End Performance

Interviewer: "What steps would you take to improve the performance of a web application that is loading slowly on the client side?"

Good Developer:
"I would start by minimizing the size of CSS and JavaScript files through minification and concatenation. I’d also look into lazy loading images and using a content delivery network (CDN) to serve static assets."

Great Developer:
"First, I’d conduct a performance audit using tools like Google Lighthouse or WebPageTest to identify the main bottlenecks. Optimizations would include reducing file sizes through minification, concatenation, and compression (e.g., Gzip or Brotli), as well as ensuring efficient use of CSS and JavaScript by eliminating unused code and deferring non-critical scripts. Implementing lazy loading for images and other resources, using a CDN for faster asset delivery, and leveraging browser caching would also be key strategies. Beyond these, I’d consider optimizing rendering performance by reducing reflows and repaints, utilizing asynchronous JavaScript loading, and considering service workers for caching assets in progressive web apps (PWAs). Finally, I’d set up ongoing monitoring of performance metrics to ensure the site continues to meet performance goals."

Tips for Answering Technical Questions in Interviews

Understand the Problem First: Before jumping to a solution, ask a few questions to clarify the problem. This helps you give a more focused answer!
Consider Multiple Angles: Consider the problem from different angles—what quick fixes can help, what long-term solutions are possible, and what trade-offs should you keep in mind? This shows a deeper understanding!
Explain Your Thought Process: Rather than just giving a solution, walk through your reasoning. Interviewers are often more interested in how you think than in whether you know the exact answer.
Discuss Both Quick Wins and Long-Term Solutions: A great answer blends quick fixes with long-term improvements, showing you care about both immediate needs and the future!
Mention Monitoring and Maintenance: Emphasizing ongoing monitoring and maintenance shows you know that fixing a problem isn’t just about going live!
Be Honest About Trade-Offs: Acknowledge that every solution has its trade-offs. Discussing these shows that you are aware of the complexities involved in technical decisions.

By following these tips, you can turn a good answer into a great one, positioning yourself as a thoughtful, knowledgeable, and thorough developer in technical interviews.

Creating Custom Hooks in React and TypeScript 👨‍💻

Majd-sufyan — Wed, 04 Jan 2023 22:16:10 +0000

React is a popular JavaScript library for building user interfaces, and TypeScript is a popular superset of JavaScript that adds optional static typing and other features. Together, React and TypeScript can provide a powerful toolset for building scalable and maintainable web applications.

One of the key features of React is its ability to reuse code through the use of custom hooks. Custom hooks are functions that allow you to extract component logic into reusable functions. They are a powerful way to share logic across your application, and can help to reduce the complexity of your code.

To create a custom hook in React and TypeScript, you will need to follow a few steps.

First, create a new file in your project and name it useMyHook.ts. This file will contain the code for your custom hook.

Next, define the function signature for your custom hook. A custom hook should always begin with the word "use" and should follow the rules for naming functions in TypeScript. For example:

import { useState, useEffect } from 'react';

export function useMyHook(): [number, (n: number) => void] {
  const [count, setCount] = useState(0);

  useEffect(() => {
    console.log(`count is ${count}`);
  }, [count]);

  return [count, setCount];
}

In this example, the custom hook is named useMyHook and it returns a tuple containing the current value of count and a function for updating count. The hook also uses the useState and useEffect hooks from React, which are used to manage state and side effects in a functional component.

To use your custom hook in a React component, import it into the component file and use it like any other hook. For example:

import { useMyHook } from './useMyHook';

function MyComponent() {
  const [count, setCount] = useMyHook();

  return (
    <div>
      <p>{count}</p>
      <button onClick={() => setCount(count + 1)}>Increment</button>
    </div>
  );
}

This component uses the useMyHook custom hook to manage the count state and render a button that increments the count when clicked.

Custom hooks are a powerful way to share logic across your React components and can help to reduce the complexity of your code. By following the steps outlined above, you can easily create custom hooks in React and TypeScript and take advantage of the benefits they offer.

React Cheatsheet for 2023 ✍️

Majd-sufyan — Sat, 31 Dec 2022 13:12:27 +0000

Welcome to my React cheat sheet! In this article, I'll cover all the common concepts and techniques we use every day when developing in React. I'll be covering topics such as React elements, element attributes, element styles, fragments, components, props, children props, conditionals, lists, context, and hooks.

Whether you're an experienced developer or new to React, this cheat sheet aims to provide you with a quick reference for all the key concepts and techniques you need to know to build powerful and scalable applications with React.

So let's get started and dive into my React cheatsheet!

React Element Attributes

JSX requires a different syntax for its attributes.

Since JSX is really JavaScript and JavaScript uses a camelcase naming convention (that is, “camelCase”), attributes are written differently than HTML.

The most common example is the class attribute, which we write as className.

<div className="App"></div>

React Element Styles

In addition to attributes, elements can also have inline styles applied to them. To set inline styles in React, you'll need to use the style attribute and pass in an object containing the desired styles. For example:

<div style={{color: 'red', fontSize: '16px'}}>Hello World</div>

React Fragments

Sometimes, you may want to render a group of elements without adding an extra DOM node around them. In these cases, you can use React fragments, which allow you to wrap multiple elements in a single container without adding any extra markup. To use a fragment, you can use the <> syntax:

<>
  <div>First element</div>
  <div>Second element</div>
</>

React Components

Components are the building blocks of a React application and allow you to split your UI into reusable pieces. You can create a component by defining a JavaScript function or class that returns a React element. For example:

function MyButton(props) {
  return <button>{props.label}</button>;
}

React Props

Props (short for "properties") are a way for components to accept data from their parent components. When a parent component renders a child component, it can pass data to the child component through props. For example:

<MyButton label="Click me" />

React Children Props

Sometimes, you may want to render a component's children inside the component itself. To do this, you can use the props.children property, which represents the content between the opening and closing tags of a component. For example:

function MyContainer(props) {
  return (
    <div>
      {props.children}
    </div>
  );
}

<MyContainer>
  <p>Content goes here</p>
</MyContainer>

React Conditionals

React allows you to use JavaScript expressions to conditionally render elements based on certain conditions. For example, you can use the && operator to only render an element if a certain condition is true:

{isLoggedIn && <p>Welcome back!</p>}

React Lists

React provides a map function that allows you to render a list of elements based on an array of data. For example:

const items = ['item 1', 'item 2', 'item 3'];

<ul>
  {items.map((item) => (
    <li>{item}</li>
  ))}
</ul>

React Context

React Context is a way to pass data through the component tree without having to pass props down manually at every level. It is often used for global data that needs to be accessible by multiple components, such as a user's theme preference or a language setting.

To create a context, you can use the createContext function and provide a default value for the context. Then, you can use the Provider component to provide the context value to its children, and the useContext hook to consume the context value from a component.

React Hooks

React Hooks are a new addition to React that allow you to use state and other features in functional components, rather than just class-based components. There are several built-in hooks that provide common functionality, such as managing state or performing side effects.

I am going to cover the 6 essential hooks you absolutely need to know:

useState
useEffect
useRef
useContext
useCallback
useMemo

React `useState` Hook

The useState hook allows you to add state to a functional component. It returns an array with two elements: the current state value and a function to update it. You can use the hook like this:

import { useState } from 'react';

function MyComponent() {
  const [count, setCount] = useState(0);

  return (
    <div>
      <p>{count}</p>
      <button onClick={() => setCount(count + 1)}>Increment</button>
    </div>
  );
}

React `useEffect` Hook

The useEffect hook allows you to perform side effects in a functional component, such as making an API call or setting up a subscription. It takes a function as an argument that will be called after the component renders. You can use the hook like this:

import { useEffect } from 'react';

function MyComponent() {
  useEffect(() => {
    // Perform side effect here
  });

  return <div>My component</div>;
}

React `useRef` Hook

The useRef hook allows you to create a mutable reference to a DOM element or value. It returns a mutable ref object whose current property is initialized to the provided argument (or null if no argument is given). You can use the hook like this:

import { useRef } from 'react';

function MyComponent() {
  const inputRef = useRef(null);

  return (
    <div>
      <input ref={inputRef} />
    </div>
  );
}

React `useContext` Hook

The useContext hook allows you to consume a context value from a functional component. It takes the context object as an argument and returns the current context value. You can use the hook like this:

import { useContext } from 'react';

const MyContext = createContext(defaultValue);

function MyComponent() {
  const value = useContext(MyContext);

  return <div>{value}</div>;
}

React `useCallback` Hook

The useCallback hook returns a memoized callback function that only changes if one of the dependencies has changed. It can be used to optimize the performance of components that rely on expensive functions or callbacks. You can use the hook like this:

import { useCallback } from 'react';

function MyComponent() {
  const callback = useCallback(() => {
   // Expensive function or callback
  }, [dependency1, dependency2]);

  return <div onClick={callback}>Click me</div>;
}

React `useMemo` Hook

The useMemo hook returns a memoized value that only changes if one of the dependencies has changed. It can be used to optimize the performance of expensive calculations by only re-computing the value when necessary. You can use the hook like this:

import { useMemo } from 'react';

function MyComponent() {
  const value = useMemo(() => {
    // Expensive calculation
  }, [dependency1, dependency2]);

  return <div>{value}</div>;
}

Conclusion

In this article, we provided a cheat sheet for some of the most common techniques and concepts used in React development, including element attributes, styles, fragments, components, props, conditionals, lists, context, and hooks. By familiarizing yourself with these concepts, you'll be well-equipped to build powerful and scalable applications with React.

You should know about the filter(Boolean) trick 💡

Majd-sufyan — Sat, 31 Dec 2022 12:23:09 +0000

The Array.prototype.filter() method in JavaScript allows you to create a new array with only the elements that pass a certain test. The filter() method takes a callback function as an argument, which should return a Boolean value indicating whether or not the element should be included in the new array.

One use case for the filter() method is to remove falsy values from an array. You can do this by using the Boolean function as the callback for the filter() method. The Boolean function returns true for values that are "truthy" and false for values that are "falsy". In JavaScript, the following values are considered falsy:

false
0 (zero)
'' (empty string)
null
undefined
NaN (Not a Number)

Here is an example of how to use the filter(Boolean) trick to remove falsy values from an array:

const values = [0, 1, '', 'hello', null, undefined, false, true, NaN];

const truthyValues = values.filter(Boolean);

console.log(truthyValues); // [1, 'hello', true]

In this example, the filter() method is called on the values array and passed the Boolean function as the callback. The filter() method will then iterate through each element in the values array and include any elements for which the Boolean function returns true in the new truthyValues array.

Here is another example of using the filter(Boolean) trick to remove falsy values from an array of objects:

const users = [
  { id: 1, name: 'John', age: 30 },
  { id: 2, name: 'Jane', age: 25 },
  { id: 3, name: 'Bill', age: 35 },
  { id: 4, name: 'Mary', age: 40 },
  { id: 5, name: '', age: null }
];

const validUsers = users.filter(user => Boolean(user.name) && Boolean(user.age));

console.log(validUsers);
/* 
[
  { id: 1, name: 'John', age: 30 },
  { id: 2, name: 'Jane', age: 25 },
  { id: 3, name: 'Bill', age: 35 },
  { id: 4, name: 'Mary', age: 40 }
]
*/

In this example, the filter() method is used to remove any users who have an empty string for their name or a null value for their age. The filter() method is passed a callback function that uses the Boolean function to check if the name and age properties of each user are truthy. The filter() method will then include only the users for which the callback function returns true in the new validUsers array.

The filter(Boolean) trick is a useful technique for removing falsy values from an array in JavaScript. It can be a powerful way to filter and transform data to meet your specific needs.

I hope you will find this trcik helpful and will help you in your journey. Feel free to give me suggestions.

Take care developer 👋