DEV Community: CodeWithDhanian

Circuit Breaker Pattern in System Design

CodeWithDhanian — Fri, 03 Apr 2026 08:07:13 +0000

In distributed systems and microservices architectures, failures are inevitable. Network latency, service overload, database slowdowns, or third-party API outages can quickly cascade into widespread system instability. The Circuit Breaker Pattern serves as a critical resilience mechanism that prevents these cascading failures by intelligently isolating faulty components. Inspired by electrical circuit breakers that interrupt current flow during overloads, the software version acts as a protective proxy around remote calls, allowing systems to fail fast, degrade gracefully, and recover automatically.

This pattern is essential for building robust, highly available applications where services depend on each other across network boundaries. By monitoring failure rates and response times, the circuit breaker stops repeated attempts to reach an unhealthy service, giving it time to recover while providing immediate feedback or fallback responses to callers.

Understanding the Circuit Breaker Pattern

The Circuit Breaker Pattern functions as a stateful wrapper around operations that interact with external services or resources. Instead of allowing every request to reach a failing downstream service—which could overwhelm it further and degrade the entire system—the circuit breaker tracks metrics such as error counts, latency, or exceptions. When failure thresholds are breached, it “trips” and redirects traffic away from the problematic service.

Key benefits include:

Prevention of cascading failures
Reduction in resource consumption on both caller and callee sides
Faster response times through immediate failure detection
Graceful degradation via fallback mechanisms
Automatic recovery without manual intervention

The pattern works best when combined with complementary techniques such as retries with exponential backoff, timeouts, rate limiting, and bulkhead isolation.

The Three States of a Circuit Breaker

A circuit breaker maintains one of three distinct states, each dictating how incoming requests are handled. These states form a finite state machine that transitions based on observed behavior and configurable thresholds.

Closed State:

This is the normal operating state. All requests pass through to the protected service. The circuit breaker monitors outcomes, counting failures within a sliding time window or consecutive failure count. If the failure rate or count exceeds a predefined threshold (for example, 50% errors in the last 10 seconds or 5 consecutive failures), the breaker transitions to the Open state. Successes reset or decrement failure counters.

Open State:

When the circuit is open, the breaker immediately rejects all requests without forwarding them to the downstream service. This prevents further load on the failing component and avoids long timeouts or resource exhaustion. Instead, the caller receives an immediate exception or a fallback response. A timeout timer (reset timeout) starts, after which the breaker moves to the Half-Open state to test recovery.

Half-Open State:

This transitional state allows a limited number of test requests (often just one or a small configurable count) to reach the service. If these probe requests succeed, the circuit breaker assumes recovery and returns to the Closed state, resetting failure counters. If any test fails, the breaker reverts to the Open state and restarts the timeout period. This cautious probing ensures the service has truly stabilized before resuming full traffic.

These state transitions enable self-healing while protecting system stability.

Detailed Implementation of the Circuit Breaker Pattern

Implementing a circuit breaker from scratch requires careful handling of concurrency, metrics tracking, and state persistence. In production, developers typically use battle-tested libraries such as Resilience4j (Java), Hystrix (legacy Java), Polly (.NET), or pybreaker (Python). Below are complete, illustrative code structures.

Pseudocode for a Generic Circuit Breaker

class CircuitBreaker {
    enum State { CLOSED, OPEN, HALF_OPEN }

    State currentState = CLOSED;
    int failureCount = 0;
    int successCount = 0;
    long lastFailureTime = 0;
    Configuration config;  // failureThreshold, timeout, successThreshold, etc.

    Object execute(Callable operation) {
        if (currentState == OPEN) {
            if (isTimeoutExpired()) {
                transitionTo(HALF_OPEN);
            } else {
                return invokeFallback();  // or throw CircuitOpenException
            }
        }

        try {
            Object result = operation.call();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure(e);
            return invokeFallback();
        }
    }

    private void onSuccess() {
        failureCount = 0;
        successCount++;
        if (currentState == HALF_OPEN && successCount >= config.successThreshold) {
            transitionTo(CLOSED);
        }
    }

    private void onFailure(Exception e) {
        failureCount++;
        lastFailureTime = currentTime();
        if (failureCount >= config.failureThreshold || currentState == HALF_OPEN) {
            transitionTo(OPEN);
        }
    }

    private boolean isTimeoutExpired() {
        return (currentTime() - lastFailureTime) > config.resetTimeout;
    }

    private void transitionTo(State newState) {
        currentState = newState;
        // Log state change, notify monitoring system
        if (newState == HALF_OPEN) {
            successCount = 0;
        }
    }

    private Object invokeFallback() {
        // Execute fallback logic, e.g., return cached data or default value
        return defaultResponse();
    }
}

Java Example Using Resilience4j Style (Conceptual Full Structure)

// Configuration
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)           // Open if failure rate > 50%
    .waitDurationInOpenState(Duration.ofSeconds(30))  // Reset timeout
    .permittedNumberOfCallsInHalfOpenState(3)         // Test calls
    .slidingWindowSize(10)              // Window for metrics
    .build();

CircuitBreaker circuitBreaker = CircuitBreaker.of("paymentService", config);

// Decorator usage
Supplier<String> decoratedSupplier = CircuitBreaker.decorateSupplier(
    circuitBreaker, 
    () -> callPaymentService()  // remote call
);

// With fallback
String result = Try.ofSupplier(decoratedSupplier)
    .recover(throwable -> fallbackPaymentResponse())
    .get();

Python Example Using a Simple Custom Implementation

import time
from enum import Enum
from typing import Callable, Any

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, failure_threshold: int = 5, reset_timeout: int = 30, success_threshold: int = 2):
        self.state = CircuitState.CLOSED
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.success_threshold = success_threshold
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = 0

    def call(self, func: Callable, *args, **kwargs) -> Any:
        if self.state == CircuitState.OPEN:
            if time.time() - self.last_failure_time > self.reset_timeout:
                self.state = CircuitState.HALF_OPEN
                self.success_count = 0
            else:
                raise CircuitBreakerOpenException("Circuit breaker is OPEN")

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise e  # or handle with fallback

    def _on_success(self):
        self.failure_count = 0
        self.success_count += 1
        if self.state == CircuitState.HALF_OPEN and self.success_count >= self.success_threshold:
            self.state = CircuitState.CLOSED

    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold or self.state == CircuitState.HALF_OPEN:
            self.state = CircuitState.OPEN

class CircuitBreakerOpenException(Exception):
    pass

These implementations highlight essential elements: configurable thresholds, state management, fallback execution, and safe transitions. In real systems, thread-safety (using locks or atomic operations) and integration with monitoring tools like Prometheus are mandatory.

When and How to Use the Circuit Breaker Pattern

Apply the Circuit Breaker Pattern to any synchronous or asynchronous call to external services, databases, or third-party APIs where failure could propagate. Common scenarios include microservices communication, payment gateways, inventory checks, or recommendation engines.

Best practices:

Combine with timeouts to avoid indefinite waits.
Implement meaningful fallbacks—cached data, default values, or queued operations.
Monitor state transitions and metrics for observability.
Tune thresholds based on service characteristics and traffic patterns.
Ensure idempotency for operations that may be retried.

The pattern shines in high-traffic environments but adds slight overhead in normal operation due to metric collection. For extremely latency-sensitive paths, evaluate whether the protection justifies the cost.

Mastering the Circuit Breaker Pattern equips system designers with a powerful tool to build resilient, fault-tolerant distributed systems that maintain availability even when individual components fail.

System Design Handbook

For more in-depth insights and comprehensive coverage of system design topics, consider purchasing the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. It will equip you with the knowledge to master complex distributed systems.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Rate Limiting & Throttling in System Design

CodeWithDhanian — Fri, 03 Apr 2026 07:54:26 +0000

In large-scale distributed systems and microservices architectures, uncontrolled incoming traffic can quickly lead to resource exhaustion, degraded performance, or complete service outages. Rate limiting and throttling serve as critical defensive mechanisms that protect backend services, ensure fair usage among clients, prevent abuse, and maintain overall system stability under varying load conditions. These techniques control the flow of requests to APIs, databases, or other resources, allowing systems to operate reliably even during traffic spikes or malicious attacks.

Understanding Rate Limiting

Rate limiting is a technique that enforces a strict upper bound on the number of requests a client, user, IP address, or API key can make within a defined time window. The primary goals include protecting against DDoS attacks, ensuring fair resource allocation, enforcing business quotas, and preventing any single client from monopolizing shared resources.

When a request exceeds the allowed limit, the system typically rejects it immediately and returns an HTTP 429 Too Many Requests status code, often accompanied by headers such as Retry-After to inform the client when it may retry.

Rate limiting operates at multiple layers: at the API gateway, within individual microservices, at the load balancer, or even at the edge using content delivery networks. In distributed environments, the rate limiter must maintain consistent state across multiple nodes, typically using a centralized store such as Redis.

Understanding Throttling

Throttling differs from rate limiting by focusing on controlling the processing speed or flow of requests rather than imposing a hard rejection limit. Instead of outright denying excess requests, throttling slows down, queues, or paces the handling of requests to maintain a steady load on the system.

While rate limiting answers the question “Is this request allowed?”, throttling addresses “How fast should this request be processed?”. Throttling is particularly useful for smoothing bursty traffic, protecting downstream services with their own rate limits, or gracefully handling temporary overload without dropping legitimate requests.

Common throttling strategies include introducing artificial delays, queuing requests in message queues, or dynamically reducing the processing rate based on current system metrics such as CPU usage or queue length.

Key Differences Between Rate Limiting and Throttling

Rate limiting provides a hard cap and immediate rejection for excess requests, making it ideal for quota enforcement and abuse prevention. Throttling prioritizes smoothing traffic and improving user experience by avoiding abrupt denials, often at the cost of increased latency for some requests. Many production systems combine both: rate limiting at the entry point for protection and throttling internally for traffic shaping.

Common Rate Limiting Algorithms

Several well-established algorithms exist for implementing rate limiting, each offering different trade-offs in terms of burst tolerance, accuracy, memory usage, and implementation complexity.

Token Bucket Algorithm

The token bucket algorithm is one of the most widely adopted approaches due to its flexibility and ability to handle controlled bursts. It models capacity as a bucket that accumulates tokens at a constant refill rate up to a maximum capacity. Each incoming request consumes one token. If tokens are available, the request is allowed; otherwise, it is rejected.

Key parameters:

Refill rate (r): Tokens added per unit time (e.g., 10 tokens per second).
Bucket capacity (b): Maximum number of tokens the bucket can hold, determining burst size.

This algorithm allows short bursts up to the bucket capacity while enforcing the long-term average rate. It is particularly suitable for public APIs where users may send occasional bursts of requests after periods of inactivity.

Complete Token Bucket Implementation Example Using Redis (Lua Script for Atomicity)

-- Token Bucket Lua Script for Redis
local key = KEYS[1]                  -- e.g., "rate:limit:user:123"
local now = tonumber(ARGV[1])        -- current timestamp in seconds
local refill_rate = tonumber(ARGV[2]) -- tokens per second
local capacity = tonumber(ARGV[3])   -- max bucket size
local tokens_requested = tonumber(ARGV[4]) or 1

-- Get current tokens and last refill time
local last_refill = tonumber(redis.call("HGET", key, "last_refill") or now)
local tokens = tonumber(redis.call("HGET", key, "tokens") or capacity)

-- Calculate new tokens to add
local elapsed = now - last_refill
local new_tokens = math.floor(elapsed * refill_rate)
tokens = math.min(tokens + new_tokens, capacity)

-- Check if enough tokens available
if tokens >= tokens_requested then
    tokens = tokens - tokens_requested
    redis.call("HSET", key, "tokens", tokens)
    redis.call("HSET", key, "last_refill", now)
    redis.call("EXPIRE", key, 3600)  -- expire after 1 hour for cleanup
    return {1, tokens}               -- allowed, remaining tokens
else
    return {0, tokens}               -- rejected, remaining tokens
end

This Lua script ensures atomic execution, preventing race conditions in distributed systems. The client calls this script via EVAL or EVALSHA commands in Redis.

Leaky Bucket Algorithm

The leaky bucket algorithm treats requests as water pouring into a bucket with a small hole at the bottom. Requests enter the bucket and are processed (leaked) at a constant fixed rate. If the bucket overflows, incoming requests are rejected or queued.

Leaky bucket excels at smoothing traffic to a steady output rate, making it ideal for scenarios requiring predictable load, such as payment processing or integration with external services that have strict rate limits. Unlike token bucket, it does not permit large bursts; excess requests are either delayed or dropped.

Fixed Window Counter Algorithm

The fixed window algorithm divides time into fixed intervals (e.g., one minute or one hour) and counts the number of requests within each window. A counter is incremented for every allowed request. When the counter exceeds the limit for the current window, further requests are rejected until the next window begins.

This approach is simple and memory-efficient but suffers from the boundary burst problem: clients can send twice the allowed rate at window edges (e.g., 100 requests at the end of one minute and another 100 immediately at the start of the next).

Simple Fixed Window Pseudocode

function isAllowed(clientId, limit, windowSeconds):
    currentWindow = floor(currentTime / windowSeconds)
    counterKey = "rate:" + clientId + ":" + currentWindow
    count = redis.INCR(counterKey)
    if count == 1:
        redis.EXPIRE(counterKey, windowSeconds)
    return count <= limit

Sliding Window Algorithms

Sliding window approaches provide higher accuracy by using a continuously moving time frame instead of rigid boundaries.

Sliding Window Log: Maintains a sorted list or set of timestamps for every request made by a client within the window. On each request, remove old timestamps outside the window and check if the remaining count is below the limit. This offers precise control but consumes significant memory for high-traffic clients.

Sliding Window Counter: A hybrid that combines fixed windows with mathematical adjustment. It tracks counts in the current and previous windows and calculates a weighted count for the sliding period. This balances accuracy and memory usage effectively.

Distributed Rate Limiting Considerations

In microservices or multi-node deployments, a single in-memory rate limiter is insufficient. Designers must ensure consistency across instances using a shared distributed cache such as Redis, Memcached, or a dedicated rate-limiting service.

Consistent hashing can route requests for the same client to the same shard, while Lua scripts or atomic operations guarantee correctness under concurrency. For extremely high scale, consider Redis Cluster or consistent hashing combined with local caching for hot clients.

Idempotency and proper error handling are essential: clients should receive clear rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset) to adjust their behavior gracefully.

Best Practices for Implementing Rate Limiting & Throttling

Apply rate limiting at multiple levels: edge (CDN or API gateway), service level, and database level. Choose the algorithm based on requirements — token bucket for burst-tolerant APIs, leaky bucket for traffic shaping, and sliding window counter for strict fairness with good performance.

Use Redis with Lua scripts for atomicity in distributed setups. Always return informative headers and consider adaptive rate limiting that dynamically adjusts limits based on system load. Combine with circuit breakers, bulkheads, and monitoring (Prometheus, Grafana) to detect and respond to abuse patterns.

For throttling, integrate with message queues (Kafka, RabbitMQ) to queue excess requests or apply exponential backoff and jitter on retries.

Rate limiting and throttling form foundational resilience patterns in system design. Proper implementation protects services, improves user experience, and enables sustainable scaling of distributed systems.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Idempotency in System Design

CodeWithDhanian — Fri, 03 Apr 2026 07:05:37 +0000

In the realm of distributed systems and microservices architectures, ensuring that operations produce the same result regardless of how many times they are executed stands as a fundamental requirement for building reliable and fault-tolerant applications. Idempotency addresses this need by guaranteeing that repeated invocations of the same operation yield identical outcomes without causing unintended side effects. This concept becomes especially critical when dealing with network failures, retries, message queues, and distributed transactions, where the same request may arrive multiple times due to timeouts, duplicate deliveries, or client retries.

Understanding Idempotency

Idempotency derives from mathematics, where a function is idempotent if applying it multiple times produces the same result as applying it once. In system design, an idempotent operation ensures that executing the same request several times has the same effect as executing it exactly once.

Key characteristics of idempotent operations include:

Safe repetition: Repeating the request does not create duplicate resources, charge a customer multiple times, or alter system state unexpectedly.
No cumulative side effects: The system state after N identical requests equals the state after a single request.
Predictable outcomes: Clients can safely retry failed operations without fear of inconsistency.

Common examples of idempotent operations include:

Retrieving a resource via GET in REST APIs
Updating a resource with a complete replacement (PUT)
Deleting a resource (DELETE)
Processing a payment with a unique idempotency key

Non-idempotent operations, such as creating a new resource with POST or incrementing a counter, require special handling to prevent duplicates.

Why Idempotency Matters in Distributed Systems

Distributed systems face inherent unreliability due to network partitions, service failures, and timeout issues. Clients and intermediaries often implement retry mechanisms with exponential backoff. Without idempotency, these retries can lead to serious problems:

Duplicate orders in e-commerce platforms
Multiple charges on customer payment methods
Inconsistent inventory levels
Corrupted data in financial ledgers

Idempotency serves as a critical defense mechanism that allows safe retries, simplifies error recovery, and supports at-least-once delivery semantics commonly found in message queues like Kafka, RabbitMQ, and SQS. It forms an essential building block alongside patterns such as the Circuit Breaker, Retry & Exponential Backoff, and Saga for distributed transactions.

Implementing Idempotency with Idempotency Keys

The most robust approach to achieving idempotency involves the use of idempotency keys. A client generates a unique identifier for each logical operation and includes it in every request. The server stores the result associated with this key and reuses it for subsequent identical requests.

Core Components of Idempotency Key Implementation

Idempotency Key: A unique string or UUID generated by the client, typically tied to a specific business operation.
Storage Layer: A durable store (database table, Redis cache, or dedicated service) that records processed keys along with their outcomes.
Validation Logic: Server-side checks to detect duplicate requests and return cached responses.
Expiration Mechanism: Optional time-based cleanup of old keys to prevent unbounded storage growth.

Complete Implementation Example: Idempotent Payment API

Consider a payment processing endpoint that must remain idempotent even under heavy retry scenarios.

Server-Side Controller (Idempotent Endpoint)

@RestController
@RequestMapping("/payments")
class PaymentController {

    PaymentService paymentService;
    IdempotencyStore idempotencyStore;  // Backed by Redis or Database

    @PostMapping
    ResponseEntity<PaymentResponse> processPayment(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody PaymentRequest request) {

        // Step 1: Check for existing result
        Optional<PaymentResponse> existingResult = idempotencyStore.getResult(idempotencyKey);
        if (existingResult.isPresent()) {
            return ResponseEntity.ok(existingResult.get());  // Return cached response
        }

        // Step 2: Validate key format and business rules
        if (!isValidIdempotencyKey(idempotencyKey)) {
            throw new InvalidIdempotencyKeyException();
        }

        try {
            // Step 3: Execute the actual business logic
            PaymentResult result = paymentService.processPayment(request);

            PaymentResponse response = mapToResponse(result);

            // Step 4: Store the result atomically with the key
            idempotencyStore.storeResult(idempotencyKey, response, Duration.ofHours(24));

            return ResponseEntity.ok(response);

        } catch (Exception e) {
            // Store failure state to prevent partial retries
            idempotencyStore.storeFailure(idempotencyKey, e.getMessage());
            throw e;
        }
    }
}

Idempotency Store Implementation (Using Redis for High Performance)

class RedisIdempotencyStore implements IdempotencyStore {

    RedisTemplate<String, Object> redisTemplate;

    Optional<PaymentResponse> getResult(String key) {
        String stored = (String) redisTemplate.opsForValue().get("idempotency:" + key);
        if (stored == null) {
            return Optional.empty();
        }
        return Optional.of(deserializeResponse(stored));
    }

    void storeResult(String key, PaymentResponse response, Duration ttl) {
        String serialized = serializeResponse(response);
        redisTemplate.opsForValue().set(
            "idempotency:" + key, 
            serialized, 
            ttl
        );
    }

    void storeFailure(String key, String errorMessage) {
        redisTemplate.opsForValue().set(
            "idempotency:" + key + ":failure", 
            errorMessage, 
            Duration.ofHours(1)
        );
    }
}

This implementation ensures that even if the same request with the identical idempotency key arrives multiple times—due to network retry, load balancer duplication, or client-side retry logic—the server processes the payment operation only once and returns the same response for all subsequent calls.

Best Practices for Idempotency

Client Responsibilities:

Generate a unique idempotency key (UUID v4 recommended) per logical operation before sending the request.
Store the key locally and reuse it for all retries of the same operation.
Include the key in a standard header such as Idempotency-Key.

Server Responsibilities:

Reject requests missing a valid idempotency key for non-safe operations.
Make storage and business logic execution atomic where possible.
Use short TTLs on stored results to manage storage growth.
Design all compensating actions in Saga patterns to also be idempotent.

Idempotency in Message-Driven Systems:
When consuming from message queues, attach idempotency keys to messages. Consumers should check the key before processing and acknowledge only after successful storage of the result. This pattern supports at-least-once delivery while preventing duplicate side effects.

Idempotency in Broader System Design Patterns

Idempotency integrates deeply with other critical concepts:

In distributed transactions using the Saga pattern, every compensating transaction must be idempotent to handle repeated failure recoveries safely.
API Gateways and service meshes can enforce idempotency at the infrastructure layer.
Event-Driven Architectures rely on idempotent event handlers to maintain consistency across services.
Retry & Exponential Backoff strategies become safe only when paired with proper idempotency controls.

In high-scale systems, idempotency often combines with rate limiting, circuit breakers, and distributed caching to create resilient request pipelines.

Mastering idempotency enables system designers to build applications that gracefully handle the realities of distributed environments while maintaining data integrity and providing consistent user experiences.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Distributed Transactions (2PC, Saga) in System Design

CodeWithDhanian — Fri, 03 Apr 2026 06:52:41 +0000

In the complex landscape of modern distributed systems, maintaining data consistency across multiple independent services and databases presents one of the most challenging problems in system design. Distributed transactions provide the foundation for ensuring that operations spanning several resources either succeed completely or fail entirely, preserving the ACID properties of atomicity, consistency, isolation, and durability. This article explores two primary approaches to handling distributed transactions: the Two-Phase Commit protocol, commonly known as 2PC, and the Saga pattern. Each method addresses the coordination of long-running transactions in environments where traditional single-database transactions fall short.

What Are Distributed Transactions

A distributed transaction involves multiple participating resources, such as separate databases, microservices, or external systems, that must coordinate to achieve a unified outcome. Unlike local transactions confined to a single resource, distributed transactions must manage cross-service consistency while dealing with network latency, partial failures, and independent scaling of components.

The core requirement remains the same as in monolithic systems: the entire operation must appear atomic to the end user. If any part fails, all changes must be undone. However, achieving this in a distributed environment introduces significant complexity because each participant operates autonomously, and communication occurs over unreliable networks. System designers must therefore select protocols that balance strong consistency with availability and performance.

The Two-Phase Commit Protocol (2PC)

The Two-Phase Commit protocol, or 2PC, stands as the classic solution for achieving strong consistency in distributed transactions. Introduced in the 1970s, 2PC relies on a central coordinator and multiple participants to ensure all-or-nothing semantics across heterogeneous resources.

Core Components of 2PC

Coordinator: The central authority responsible for driving the transaction. It receives the initial transaction request and manages the voting and decision process.
Participants: The individual resources (databases or services) that perform local work and respond to the coordinator's instructions.
Transaction Manager: Often implemented using standards such as XA (eXtended Architecture) for database interactions.

Phases of the 2PC Protocol

2PC operates in two distinct phases, ensuring safety before any permanent changes occur.

Prepare Phase (Voting Phase):

The coordinator sends a prepare message to all participants. Each participant performs the necessary local operations, acquires locks, writes changes to a durable log, and responds with either ready (vote yes) or abort (vote no). If any participant votes no or fails to respond, the coordinator decides to abort.

Commit Phase (Decision Phase):

If all participants vote ready, the coordinator logs the global commit decision and sends commit messages to every participant. Each participant then applies the changes permanently and releases locks. If the decision is to abort, the coordinator sends rollback messages, and participants undo their local changes using the prepared log entries.

Pseudocode Implementation of 2PC Coordinator

class TwoPhaseCommitCoordinator {
    List<Participant> participants;
    TransactionLog log;

    void beginTransaction(Transaction tx) {
        log.write("BEGIN_TX", tx.id);
        boolean allReady = true;

        // Prepare Phase
        for each participant in participants {
            Response response = participant.prepare(tx);
            if (!response.isReady()) {
                allReady = false;
                break;
            }
        }

        // Decision
        if (allReady) {
            log.write("GLOBAL_COMMIT", tx.id);
            for each participant in participants {
                participant.commit(tx);
            }
        } else {
            log.write("GLOBAL_ABORT", tx.id);
            for each participant in participants {
                participant.rollback(tx);
            }
        }
    }
}

Pseudocode for a 2PC Participant

class DatabaseParticipant implements Participant {
    LocalDatabase db;
    UndoLog undoLog;

    Response prepare(Transaction tx) {
        try {
            db.acquireLocks(tx.operations);
            db.executeOperations(tx.operations);  // tentative changes
            undoLog.recordUndoInfo(tx);
            return new Response(true, "READY");
        } catch (Exception e) {
            return new Response(false, "ABORT");
        }
    }

    void commit(Transaction tx) {
        db.makeChangesPermanent(tx);
        db.releaseLocks(tx);
        undoLog.clear(tx);
    }

    void rollback(Transaction tx) {
        db.applyUndo(undoLog.getUndoInfo(tx));
        db.releaseLocks(tx);
        undoLog.clear(tx);
    }
}

These code structures illustrate the blocking nature of 2PC: participants hold locks from the prepare phase until the final decision arrives. The coordinator must persist its decision durably before proceeding, ensuring recoverability after crashes.

Limitations of 2PC

While 2PC guarantees strong consistency, it suffers from several critical drawbacks. The coordinator becomes a single point of failure and a performance bottleneck. The protocol is blocking—if the coordinator fails after the prepare phase, participants remain locked indefinitely until recovery. Network partitions can cause prolonged unavailability. In high-throughput microservices environments, these issues make 2PC impractical for long-lived operations.

The Saga Pattern

The Saga pattern offers a fundamentally different approach to distributed transactions by embracing eventual consistency instead of immediate strong consistency. Originally described in the 1980s for handling long-lived transactions, a Saga decomposes a large distributed transaction into a sequence of smaller, local transactions. Each local transaction has an associated compensating transaction that undoes its effects if later steps fail.

Key Principles of Saga

Local Transactions: Each service executes its part independently and commits immediately.
Compensating Transactions: Reversible operations that restore the system to a consistent state without global rollback.
No Global Lock: Resources remain available throughout the process.
Eventual Consistency: The system converges to a consistent state over time rather than instantly.

Two Implementation Styles of Saga

Choreography-Based Saga:

Services communicate directly through events. Each service listens for events from previous steps and publishes its own events upon completion or failure. No central controller exists. This style promotes loose coupling but can become difficult to trace as the number of services grows.

Orchestration-Based Saga:

A central Saga Orchestrator coordinates the flow by sending commands to services and reacting to their responses or events. The orchestrator maintains the overall state and decides the next step or triggers compensation. This approach provides clearer visibility into the transaction flow and simplifies error handling.

Complete Orchestration-Based Saga Example: E-commerce Order Processing

Consider an online store where placing an order involves three services: Order Service, Payment Service, and Inventory Service. The Saga ensures that if payment fails, inventory is not deducted, or if inventory is unavailable, payment is refunded.

Saga Orchestrator Pseudocode (Full Structure)

class OrderSagaOrchestrator {
    OrderService orderService;
    PaymentService paymentService;
    InventoryService inventoryService;
    SagaStateRepository stateRepo;

    void startOrderSaga(OrderRequest request) {
        SagaInstance saga = new SagaInstance(request.orderId);
        stateRepo.save(saga);

        // Step 1: Create Order (local transaction)
        Order order = orderService.createOrder(request);
        saga.updateStep("ORDER_CREATED", order);

        try {
            // Step 2: Process Payment
            Payment payment = paymentService.processPayment(order);
            saga.updateStep("PAYMENT_SUCCESS", payment);

            // Step 3: Reserve Inventory
            InventoryReservation reservation = inventoryService.reserveInventory(order);
            saga.updateStep("INVENTORY_RESERVED", reservation);

            saga.complete();
            return;

        } catch (PaymentFailedException e) {
            // Compensation: Cancel Order
            orderService.cancelOrder(order);
            saga.fail("PAYMENT_FAILED");
        } catch (InventoryUnavailableException e) {
            // Compensation Chain
            paymentService.refundPayment(payment);
            orderService.cancelOrder(order);
            saga.fail("INVENTORY_FAILED");
        }
    }

    // Compensating transaction examples
    void compensatePayment(Payment payment) {
        paymentService.refundPayment(payment);  // idempotent refund
    }

    void compensateOrder(Order order) {
        orderService.cancelOrder(order);  // releases any reservations
    }
}

Service-Level Local Transaction Example (Inventory Service)

class InventoryService {
    InventoryRepository repo;

    InventoryReservation reserveInventory(Order order) {
        // Local transaction - fully committed immediately
        return repo.withinTransaction(() -> {
            Stock stock = repo.findStock(order.productId);
            if (stock.quantity < order.quantity) {
                throw new InventoryUnavailableException();
            }
            stock.quantity -= order.quantity;
            repo.save(stock);
            return new InventoryReservation(order.orderId, order.quantity);
        });
    }

    // Compensating transaction - public and idempotent
    void releaseInventory(InventoryReservation reservation) {
        repo.withinTransaction(() -> {
            Stock stock = repo.findStock(reservation.productId);
            stock.quantity += reservation.quantity;
            repo.save(stock);
        });
    }
}

This full code structure demonstrates how the orchestrator drives the Saga while each service remains responsible only for its local ACID transaction and its compensating transaction. Idempotency keys should be included in every command and compensation to handle retries safely after network failures.

Advantages of the Saga Pattern

Saga excels in microservices because it avoids long-held locks, improves availability, and scales horizontally. Failures trigger targeted compensations rather than global aborts. The pattern naturally fits event-driven architectures and works seamlessly with message queues such as Kafka or RabbitMQ for reliable event delivery.

Choosing Between 2PC and Saga

2PC suits scenarios demanding immediate strong consistency, such as financial systems where partial states are unacceptable. Saga fits better for business processes that tolerate temporary inconsistencies, prioritize high availability, and involve long-running workflows across many services. In practice, many system designs combine both: 2PC for critical synchronous steps within a bounded context and Saga for cross-context orchestration.

The Saga pattern, supported by modern frameworks, has become the de facto standard for distributed transactions in cloud-native microservices due to its resilience and performance characteristics. Proper implementation requires careful design of compensating transactions, idempotency, and comprehensive monitoring of Saga instances to detect and resolve stuck workflows.

Distributed Transactions (2PC, Saga) in System Design remains a cornerstone topic that every professional system designer must master to build reliable, scalable, and maintainable large-scale applications.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Event-Driven Architecture in System Design

CodeWithDhanian — Fri, 03 Apr 2026 06:34:44 +0000

Event-Driven Architecture represents a foundational paradigm in modern system design where the entire flow of processing is triggered and governed by events rather than direct synchronous calls between components. In this approach, systems detect meaningful changes in state, encapsulate them as events, and propagate them asynchronously across decoupled services. This enables real-time responsiveness, loose coupling, and high scalability while allowing individual components to evolve independently without breaking the overall system.

What is Event-Driven Architecture

Event-Driven Architecture shifts away from the traditional request-response model commonly seen in RESTful APIs or monolithic applications. Instead of one service directly invoking another and waiting for a reply, producers emit events that signal something has occurred. Consumers then react to these events at their own pace. This asynchronous nature eliminates blocking calls, reduces latency bottlenecks, and supports massive concurrency.

At its core, Event-Driven Architecture treats events as first-class citizens. An event is an immutable record of a fact that happened in the past, carrying both metadata and payload data. Examples include OrderPlaced, UserRegistered, or PaymentProcessed. These events drive the behavior of the entire distributed system without requiring tight integration between services.

Core Components of Event-Driven Architecture

Every Event-Driven Architecture relies on four essential building blocks that work together to ensure reliable event flow.

Events

An event is the fundamental unit of communication. It must be immutable, idempotent, and self-contained. Each event typically includes a unique event ID, timestamp, event type, source, and a payload with domain-specific data. Events are never altered after creation; instead, new events represent subsequent state changes.

Producers

Producers are the components responsible for detecting state changes and publishing events to the central event broker. A producer can be a microservice, a database trigger, or an external system. Upon generating an event, the producer serializes it into a standard format such as JSON or Avro and sends it reliably to the broker.

Consumers

Consumers subscribe to one or more event streams and execute business logic when matching events arrive. A single event can be consumed by multiple consumers simultaneously, enabling parallel processing. Consumers can be grouped into consumer groups to achieve load balancing and horizontal scaling.

Event Broker

The event broker serves as the reliable messaging backbone. It receives events from producers, persists them durably, and delivers them to consumers. Popular event brokers include Apache Kafka for high-throughput streaming and RabbitMQ for flexible routing. The broker guarantees at-least-once, exactly-once, or at-most-once delivery semantics depending on configuration.

How Event-Driven Architecture Works in Practice

Consider an e-commerce platform built with Event-Driven Architecture. When a customer places an order, the Order Service acts as a producer and publishes an OrderPlaced event. This event flows to the event broker and is immediately available to three independent consumers:

The Inventory Service subtracts stock and publishes an InventoryUpdated event.
The Payment Service processes the transaction and publishes a PaymentProcessed event.
The Notification Service sends an email confirmation to the customer.

None of these services call each other directly. They remain completely decoupled, allowing each to scale, fail, or be updated independently while the event broker ensures reliable delivery.

Key Patterns in Event-Driven Architecture

Several proven patterns elevate Event-Driven Architecture from basic messaging to sophisticated system design solutions.

Event Sourcing

Event Sourcing stores the complete history of events rather than just the current state of an entity. The current state of any object is reconstructed by replaying the sequence of events from the beginning. This pattern provides perfect auditability, time-travel debugging, and easy recovery from failures.

Command Query Responsibility Segregation (CQRS)

CQRS separates the write path (commands) from the read path (queries). Commands generate events that update the write model. A separate read model is kept synchronized through event subscriptions. This allows optimized data structures for reads while maintaining strong consistency on writes.

Saga Pattern

The Saga Pattern orchestrates long-running distributed transactions without relying on traditional two-phase commit. Each step in a business process publishes a completion event or a compensating event on failure. For example, if an order fails payment, a CancelOrder event triggers compensating actions across services to maintain overall consistency.

Implementation Example Using Apache Kafka

Apache Kafka is the industry-standard event broker for high-scale Event-Driven Architecture. Below are complete, production-ready code snippets in Python using the official confluent-kafka library.

Kafka Producer Implementation

from confluent_kafka import Producer
import json
import socket

def delivery_callback(err, msg):
    if err:
        print(f"Message delivery failed: {err}")
    else:
        print(f"Message delivered to {msg.topic()} [{msg.partition()}]")

conf = {
    'bootstrap.servers': 'kafka-broker-1:9092,kafka-broker-2:9092',
    'client.id': socket.gethostname(),
    'acks': 'all',                    # Wait for all in-sync replicas
    'enable.idempotence': True        # Prevent duplicate events
}

producer = Producer(conf)

# Publish an OrderPlaced event
event_data = {
    "event_id": "evt-1234567890",
    "event_type": "OrderPlaced",
    "timestamp": "2026-04-03T06:18:00Z",
    "payload": {
        "order_id": "ORD-98765",
        "user_id": "USR-54321",
        "items": [{"product_id": "PROD-111", "quantity": 2}],
        "total_amount": 149.99
    }
}

producer.produce(
    topic='orders',
    value=json.dumps(event_data).encode('utf-8'),
    key=event_data["payload"]["order_id"],  # Ensures ordering per order
    callback=delivery_callback
)

producer.flush()  # Ensure all messages are sent before exit

This producer guarantees exactly-once semantics through idempotence and waits for full replication before considering the event published.

Kafka Consumer Implementation

from confluent_kafka import Consumer, KafkaError
import json

conf = {
    'bootstrap.servers': 'kafka-broker-1:9092,kafka-broker-2:9092',
    'group.id': 'inventory-service-group',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,      # Manual commit for exactly-once
    'isolation.level': 'read_committed'
}

consumer = Consumer(conf)
consumer.subscribe(['orders'])

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is None:
        continue
    if msg.error():
        if msg.error().code() == KafkaError._PARTITION_EOF:
            continue
        else:
            print(f"Error: {msg.error()}")
            break

    event = json.loads(msg.value().decode('utf-8'))

    if event["event_type"] == "OrderPlaced":
        # Process inventory deduction
        print(f"Processing inventory for order {event['payload']['order_id']}")
        # ... business logic here ...

        # Publish follow-up event
        # (In real systems this would use a separate producer)

        # Manual commit only after successful processing
        consumer.commit(msg)

The consumer belongs to a consumer group, processes events in order within each partition, and commits offsets only after successful business logic execution.

Challenges in Event-Driven Architecture

While powerful, Event-Driven Architecture introduces specific complexities that must be addressed:

Eventual Consistency: Data across services may temporarily differ until all events propagate.
Event Ordering: Guaranteeing strict chronological order requires careful partitioning and key selection.
Idempotency: Consumers must handle duplicate events gracefully using event IDs or deduplication tables.
Debugging Distributed Flows: Tracing a single business transaction across dozens of events requires distributed tracing tools.
Schema Evolution: Events must support forward and backward compatibility through schema registries.

Best Practices for Event-Driven Architecture

To build robust Event-Driven Architecture systems, always:

Design events as facts about the past, never as commands.
Use Avro or Protobuf with a schema registry for type safety.
Implement dead-letter queues for failed events.
Monitor event lag, throughput, and consumer health continuously.
Version events explicitly and maintain backward compatibility.
Combine Event-Driven Architecture with CQRS and Event Sourcing only when business requirements justify the added complexity.

Event-Driven Architecture empowers system design teams to create resilient, scalable, and maintainable distributed systems that respond instantly to real-world changes.

System Design Handbook

To master every concept in system design including Event-Driven Architecture, purchase the complete System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf.

Buy me coffee to support my content at https://ko-fi.com/codewithdhanian.

Message Queues (Kafka, RabbitMQ, SQS) in System Design

CodeWithDhanian — Fri, 03 Apr 2026 06:09:40 +0000

Introduction to Message Queues in System Design

Message queues serve as the backbone of asynchronous communication in distributed systems. They enable producers to send messages that consumers process independently, without requiring both parties to be available simultaneously. This decoupling eliminates tight temporal dependencies, buffers traffic during spikes, and provides fault tolerance by allowing retries and dead-letter queues for failed messages.

In system design, message queues address critical challenges such as scalability, reliability, and throughput. They support patterns including event-driven architecture, microservices communication, task queues, and log aggregation. By persisting messages durably, they ensure data survives failures and can be replayed when necessary. Message queues come in two broad categories: traditional task queues focused on routing and delivery, and event streams optimized for high-volume, ordered, replayable data.

Three prominent implementations dominate modern system design: Apache Kafka for high-throughput event streaming, RabbitMQ for flexible routing and messaging patterns, and Amazon SQS for fully managed, serverless queue operations. Each offers distinct strengths in architecture, delivery guarantees, and operational model.

Apache Kafka: Distributed Event Streaming Platform

Apache Kafka functions as a distributed event streaming platform rather than a simple message queue. It excels in scenarios demanding massive throughput, durability, and replayability across thousands of producers and consumers.

Core Architecture of Kafka

A Kafka cluster consists of multiple brokers that store and serve data. Topics act as logical categories for messages, each divided into partitions for parallelism. Every partition is an ordered, immutable log stored on a broker.

Replication ensures fault tolerance: each partition has a leader handling reads and writes, plus followers (in-sync replicas) that copy data. If the leader fails, a follower is elected. Producers write to topics, while consumers read from partitions and commit offsets to track progress. Consumer groups enable load balancing, with each partition assigned to exactly one consumer in the group.

Kafka achieves exactly-once semantics through idempotent producers (using sequence numbers to deduplicate retries) and transactions (atomic produce-and-commit operations). Modern deployments use KRaft mode for metadata management via the Raft consensus protocol, eliminating external coordination dependencies.

Kafka delivers high throughput via batching, compression, and zero-copy transfers. It supports log compaction for stateful streams and integrates seamlessly with stream processing frameworks.

Producer and Consumer Implementation in Kafka

Here is a complete Python implementation using the kafka-python library for a basic producer and consumer. This demonstrates idempotent production and offset management.

from kafka import KafkaProducer, KafkaConsumer
from kafka.errors import KafkaError
import json
import time

# Producer configuration
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    acks='all',                    # Wait for all in-sync replicas
    retries=3,                     # Retry on transient failures
    enable_idempotence=True,       # Prevent duplicates on retries
    batch_size=16384,              # Batch records for efficiency
    linger_ms=5                    # Wait briefly to fill batches
)

# Send messages
for i in range(100):
    message = {'event_id': i, 'data': f'payload-{i}'}
    future = producer.send('user-events', value=message, key=str(i).encode())
    try:
        record_metadata = future.get(timeout=10)
        print(f"Message sent to topic {record_metadata.topic} partition {record_metadata.partition} offset {record_metadata.offset}")
    except KafkaError as e:
        print(f"Failed to send: {e}")
producer.flush()
producer.close()

Explanation of producer code: The bootstrap_servers connects to the cluster. acks='all' ensures durability by waiting for replication. enable_idempotence=True guarantees no duplicates. Batching via batch_size and linger_ms maximizes throughput. The send method returns a future for asynchronous confirmation.

# Consumer configuration
consumer = KafkaConsumer(
    'user-events',
    bootstrap_servers=['localhost:9092'],
    group_id='event-processors',
    value_deserializer=lambda x: json.loads(x.decode('utf-8')),
    auto_offset_reset='earliest',   # Start from beginning if no offset
    enable_auto_commit=False        # Manual control for exactly-once
)

for message in consumer:
    print(f"Consumed: {message.value} from partition {message.partition} offset {message.offset}")
    # Process message here
    # On success: consumer.commit() for manual offset commit

Explanation of consumer code: The group_id joins a consumer group for parallel processing. auto_offset_reset controls initial position. Manual commits allow transactional exactly-once processing when combined with producer transactions. Offsets are committed only after successful handling.

RabbitMQ: Flexible Messaging Broker with Advanced Routing

RabbitMQ implements the AMQP protocol as a robust message broker optimized for complex routing and reliable delivery. It suits task queues, work distribution, and scenarios requiring sophisticated message patterns.

Core Architecture of RabbitMQ

RabbitMQ uses exchanges to route messages to queues based on bindings and routing keys. Producers publish to exchanges; consumers pull from queues.

Four main exchange types exist:

Direct: Routes based on exact routing key match.
Fanout: Broadcasts to all bound queues (ignores key).
Topic: Uses pattern matching on routing keys (e.g., user.*.event).
Headers: Routes based on message header attributes.

Queues can be durable (survive restarts), exclusive, or auto-delete. Acknowledgments ensure reliable delivery: consumers explicitly acknowledge after processing. Prefetch limits unacknowledged messages per consumer for flow control. Dead-letter queues capture failed messages.

RabbitMQ supports clustering for high availability and mirrored queues for replication. It provides low-latency delivery and works across multiple protocols.

Producer and Consumer Implementation in RabbitMQ

Here is a complete Python implementation using the pika library.

import pika
import json

# Establish connection and channel
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare durable queue and direct exchange
channel.exchange_declare(exchange='user-events', exchange_type='direct', durable=True)
channel.queue_declare(queue='event-processor', durable=True)
channel.queue_bind(exchange='user-events', queue='event-processor', routing_key='user.created')

# Producer: publish message
for i in range(100):
    message = {'event_id': i, 'type': 'user.created', 'data': f'payload-{i}'}
    channel.basic_publish(
        exchange='user-events',
        routing_key='user.created',
        body=json.dumps(message),
        properties=pika.BasicProperties(
            delivery_mode=2,  # Persistent
        )
    )
    print(f"Published event {i}")

connection.close()

Explanation of producer code: The exchange and queue are declared durable for persistence. basic_publish with delivery_mode=2 ensures the message survives broker restarts. Routing key determines delivery via the binding.

# Consumer setup
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

channel.queue_declare(queue='event-processor', durable=True)

def callback(ch, method, properties, body):
    message = json.loads(body)
    print(f"Processed: {message}")
    ch.basic_ack(delivery_tag=method.delivery_tag)  # Manual ack

channel.basic_qos(prefetch_count=1)  # Fair dispatch
channel.basic_consume(queue='event-processor', on_message_callback=callback)

print("Waiting for messages...")
channel.start_consuming()

Explanation of consumer code: basic_qos with prefetch_count=1 prevents overload. Manual basic_ack confirms successful processing; unacknowledged messages return to the queue. This pattern supports idempotency and retry logic.

Amazon SQS: Fully Managed Serverless Queues

Amazon SQS provides a fully managed message queue service that removes infrastructure overhead. It focuses on simplicity and seamless integration within cloud-native architectures.

Core Architecture of SQS

SQS offers two queue types:

Standard queues: Deliver at-least-once with high throughput and scalability. Messages may arrive out of order or duplicated.
FIFO queues: Guarantee exactly-once processing and strict ordering within message groups.

Producers send messages via API; consumers poll for messages. Visibility timeout hides a received message temporarily, preventing concurrent processing. If not deleted within the timeout, the message reappears. Long polling waits up to 20 seconds for messages, reducing empty responses. Dead-letter queues capture messages failing after a configurable receive count.

SQS handles replication, encryption, and scaling automatically. It integrates natively with other cloud services for event-driven workflows.

Producer and Consumer Implementation in SQS

Here is a complete Python implementation using boto3.

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'  # Replace with actual URL

# Producer: send message
for i in range(100):
    message = {'event_id': i, 'data': f'payload-{i}'}
    response = sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps(message),
        MessageAttributes={
            'EventType': {
                'DataType': 'String',
                'StringValue': 'user.created'
            }
        }
    )
    print(f"Sent message ID: {response['MessageId']}")

Explanation of producer code: send_message accepts body and optional attributes for filtering. SQS handles durability and distribution automatically.

# Consumer: receive and process
while True:
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=1,
        WaitTimeSeconds=20,          # Long polling
        VisibilityTimeout=30         # Hide for 30 seconds
    )

    if 'Messages' in response:
        message = response['Messages'][0]
        body = json.loads(message['Body'])
        print(f"Received and processing: {body}")

        # Process logic here

        # Delete after success
        sqs.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message['ReceiptHandle']
        )
    else:
        print("No messages available")

Explanation of consumer code: Long polling via WaitTimeSeconds improves efficiency. Visibility timeout prevents duplicate processing. delete_message removes the message permanently using the receipt handle.

Choosing the Right Message Queue

Kafka shines for event streaming with replayability and massive scale. RabbitMQ excels in complex routing and traditional task queues. SQS offers zero-ops management for simple decoupling in cloud environments. Evaluate based on throughput needs, ordering requirements, operational burden, and delivery semantics when designing systems.

To master these and more concepts in system design, consider purchasing the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. Buy me coffee to support my content at https://ko-fi.com/codewithdhanian.

API Design (REST, GraphQL, gRPC) in System Design

CodeWithDhanian — Fri, 03 Apr 2026 05:47:43 +0000

In the intricate landscape of system design, API design serves as the foundational communication layer that enables different components, services, and clients to interact seamlessly at scale. The choice of API paradigm directly influences performance, maintainability, scalability, and developer experience. Three dominant approaches dominate modern distributed systems: REST, GraphQL, and gRPC. Each addresses distinct challenges in data exchange, efficiency, and real-time capabilities while fitting specific architectural patterns such as microservices, event-driven architectures, and high-throughput applications.

REST API Design

REST, or Representational State Transfer, is an architectural style introduced by Roy Fielding that emphasizes a stateless, client-server model built on standard HTTP protocols. In system design, REST APIs excel when simplicity, cacheability, and broad compatibility are paramount. The core constraints of REST include client-server separation, statelessness, cacheability, uniform interface, layered system, and code on demand.

A REST API treats data as resources identified by URIs. Operations on these resources map to HTTP methods: GET for retrieval, POST for creation, PUT for full updates, PATCH for partial updates, and DELETE for removal. This mapping aligns with CRUD operations while enforcing idempotency where applicable—PUT and DELETE are idempotent, whereas POST is not.

HTTP status codes provide explicit feedback: 2xx for success, 4xx for client errors, and 5xx for server errors. REST leverages HTTP headers for metadata, such as Content-Type, Authorization, and Cache-Control. Versioning is typically handled via URI paths (/v1/users), headers, or query parameters to maintain backward compatibility in evolving systems.

Here is a complete, production-ready REST API implementation using Node.js and Express to illustrate resource management for a user service in a microservices environment:

const express = require('express');
const app = express();
app.use(express.json());

// In-memory store for demonstration (replace with database in production)
let users = [{ id: 1, name: 'Alice', email: 'alice@example.com' }];

// GET all users - retrieval with optional pagination and filtering
app.get('/api/v1/users', (req, res) => {
  const { page = 1, limit = 10 } = req.query;
  const startIndex = (page - 1) * limit;
  const paginatedUsers = users.slice(startIndex, startIndex + parseInt(limit));
  res.status(200).json({
    data: paginatedUsers,
    meta: { total: users.length, page: parseInt(page), limit: parseInt(limit) }
  });
});

// GET single user by ID - resource-specific endpoint
app.get('/api/v1/users/:id', (req, res) => {
  const user = users.find(u => u.id === parseInt(req.params.id));
  if (!user) return res.status(404).json({ error: 'User not found' });
  res.status(200).json(user);
});

// POST create user - non-idempotent creation with validation
app.post('/api/v1/users', (req, res) => {
  const { name, email } = req.body;
  if (!name || !email) return res.status(400).json({ error: 'Name and email required' });
  const newUser = { id: users.length + 1, name, email };
  users.push(newUser);
  res.status(201).json(newUser);
});

// PUT full update - idempotent
app.put('/api/v1/users/:id', (req, res) => {
  const userIndex = users.findIndex(u => u.id === parseInt(req.params.id));
  if (userIndex === -1) return res.status(404).json({ error: 'User not found' });
  users[userIndex] = { id: parseInt(req.params.id), ...req.body };
  res.status(200).json(users[userIndex]);
});

// PATCH partial update
app.patch('/api/v1/users/:id', (req, res) => {
  const user = users.find(u => u.id === parseInt(req.params.id));
  if (!user) return res.status(404).json({ error: 'User not found' });
  Object.assign(user, req.body);
  res.status(200).json(user);
});

// DELETE user - idempotent removal
app.delete('/api/v1/users/:id', (req, res) => {
  const userIndex = users.findIndex(u => u.id === parseInt(req.params.id));
  if (userIndex === -1) return res.status(404).json({ error: 'User not found' });
  users.splice(userIndex, 1);
  res.status(204).send();
});

app.listen(3000, () => console.log('REST API running on port 3000'));

This snippet demonstrates stateless design—each request contains all necessary information. In a real system, integrate database indexing, caching with Redis, and rate limiting at the API gateway level for horizontal scaling.

GraphQL API Design

GraphQL is a query language and runtime for APIs created by Facebook that allows clients to request exactly the data they need in a single round trip, eliminating over-fetching and under-fetching common in REST. In system design, GraphQL shines in complex, hierarchical data models and client-driven microservices where flexibility is essential.

The GraphQL schema defines types, queries, mutations, and subscriptions using Schema Definition Language (SDL). A single endpoint (/graphql) handles all operations via POST requests containing a query or mutation document. Resolvers map fields to data-fetching logic, enabling nested queries without multiple HTTP calls.

GraphQL supports introspection for self-documentation and subscriptions over WebSockets for real-time updates. Batching and caching at the resolver level prevent N+1 query problems using tools like DataLoader.

Here is a complete GraphQL setup using Node.js with Apollo Server and an in-memory store, showcasing a user service with nested relationships:

const { ApolloServer, gql } = require('apollo-server');

// Schema definition
const typeDefs = gql`
  type User {
    id: ID!
    name: String!
    email: String!
    posts: [Post!]!
  }
  type Post {
    id: ID!
    title: String!
    content: String!
    author: User!
  }
  type Query {
    users: [User!]!
    user(id: ID!): User
  }
  type Mutation {
    createUser(name: String!, email: String!): User!
  }
  type Subscription {
    userCreated: User!
  }
`;

// Resolvers with DataLoader pattern for N+1 prevention (simplified)
const users = [{ id: '1', name: 'Alice', email: 'alice@example.com', posts: ['101'] }];
const posts = [{ id: '101', title: 'GraphQL Basics', content: 'Deep dive...', authorId: '1' }];

const resolvers = {
  Query: {
    users: () => users,
    user: (_, { id }) => users.find(u => u.id === id)
  },
  Mutation: {
    createUser: (_, { name, email }) => {
      const newUser = { id: String(users.length + 1), name, email, posts: [] };
      users.push(newUser);
      return newUser;
    }
  },
  User: {
    posts: (parent) => posts.filter(p => parent.posts.includes(p.id))
  }
};

const server = new ApolloServer({ typeDefs, resolvers });
server.listen().then(({ url }) => console.log(`GraphQL server ready at ${url}`));

This structure centralizes data fetching while giving clients precise control. In distributed systems, GraphQL integrates with API gateways for schema stitching across microservices and rate limiting per operation complexity.

gRPC API Design

gRPC is a high-performance, open-source RPC framework developed by Google that uses HTTP/2 for transport and Protocol Buffers for binary serialization. In system design, gRPC is preferred for internal service-to-service communication in polyglot microservices due to its low latency, built-in streaming, and strong typing.

gRPC contracts are defined in .proto files, which generate client and server stubs in multiple languages. It supports four communication patterns: unary, server streaming, client streaming, and bidirectional streaming. HTTP/2 multiplexing enables concurrent requests over a single connection, with automatic flow control and header compression.

Here is a complete gRPC definition and implementation example using a .proto file for a user service, followed by a Python server snippet:

syntax = "proto3";

package users;

service UserService {
  rpc GetUser (UserRequest) returns (UserResponse);
  rpc ListUsers (stream Pagination) returns (stream UserResponse);
  rpc CreateUser (UserCreateRequest) returns (UserResponse);
  rpc StreamUserUpdates (stream UserUpdate) returns (stream UserResponse);
}

message UserRequest {
  string id = 1;
}

message Pagination {
  int32 page = 1;
  int32 limit = 2;
}

message UserCreateRequest {
  string name = 1;
  string email = 2;
}

message UserResponse {
  string id = 1;
  string name = 2;
  string email = 3;
}

message UserUpdate {
  string id = 1;
  string field = 2;
  string value = 3;
}

Corresponding Python server using grpcio (full implementation):

import grpc
from concurrent import futures
import users_pb2
import users_pb2_grpc

class UserService(users_pb2_grpc.UserServiceServicer):
    def GetUser(self, request, context):
        # Simulate database lookup
        return users_pb2.UserResponse(id=request.id, name="Alice", email="alice@example.com")

    def ListUsers(self, request_iterator, context):
        for pagination in request_iterator:
            # Stream users with pagination
            for i in range(pagination.limit):
                yield users_pb2.UserResponse(id=str(i), name=f"User{i}", email=f"user{i}@example.com")

    def CreateUser(self, request, context):
        return users_pb2.UserResponse(id="new123", name=request.name, email=request.email)

    def StreamUserUpdates(self, request_iterator, context):
        for update in request_iterator:
            yield users_pb2.UserResponse(id=update.id, name="Updated", email="updated@example.com")

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
users_pb2_grpc.add_UserServiceServicer_to_server(UserService(), server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()

gRPC leverages binary format for smaller payloads and native streaming for real-time scenarios, making it ideal for low-latency distributed systems. Service discovery and load balancing integrate naturally with Kubernetes and Istio.

Choosing the Right API Design in System Design

REST prioritizes simplicity and caching for public-facing APIs where broad client support matters. GraphQL solves data flexibility challenges in frontend-heavy applications and complex data graphs. gRPC delivers superior performance and strict contracts for internal microservices communication, especially under high load or with polyglot teams.

Trade-offs include REST’s verbosity versus GraphQL’s query complexity management and gRPC’s ecosystem maturity. In a complete system, hybrid approaches are common: REST for external clients, GraphQL for mobile/web, and gRPC for backend service meshes. Security considerations—OAuth, JWT, mTLS—apply uniformly, while observability tools like distributed tracing ensure reliability across paradigms.

For a comprehensive guide that builds upon these foundations and covers the full spectrum of system design principles, purchase the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. Your purchase directly supports the creation of in-depth technical content. Buy me a coffee to support my content at: https://ko-fi.com/codewithdhanian.

Consistency Patterns (Strong, Eventual, Weak) in System Design

CodeWithDhanian — Thu, 02 Apr 2026 20:18:47 +0000

Understanding Consistency in Distributed Systems

In distributed systems, consistency defines how and when updates to data become visible across multiple nodes or replicas. When a client performs a write operation on one node, the system must decide whether subsequent read operations on any other node will immediately reflect that change or tolerate some delay. This decision directly influences availability, latency, throughput, and overall system behavior under network partitions or failures.

Consistency patterns provide structured guarantees that help architects balance these competing requirements. The three primary patterns—Strong Consistency, Eventual Consistency, and Weak Consistency—form a spectrum from the strictest guarantees to the most relaxed. Each pattern addresses different real-world demands, from financial accuracy to massive-scale user-generated content.

Strong Consistency

Strong Consistency, also known as linearizability, guarantees that once a write operation completes successfully, every subsequent read operation—regardless of which replica or client issues it—will return the most recent write value or a newer one. There is no window for stale data. All operations appear to execute in a single, global, sequential order as if on a single atomic copy of the data.

This pattern enforces immediate visibility of updates. If Client A writes a value and receives confirmation, Client B reading immediately afterward will always see the updated value. The system achieves this through tight synchronization mechanisms such as quorum-based replication, consensus protocols like Paxos or Raft, or two-phase commit (2PC) for distributed transactions.

How Strong Consistency Works in Practice

A typical architecture uses a primary node (or leader) that coordinates writes. When a write arrives:

The primary applies the change locally.
It replicates the update synchronously to a quorum of replicas (for example, a majority of nodes in a five-node cluster requires acknowledgment from at least three).
Only after the quorum confirms does the primary acknowledge success to the client.
Any read request, whether directed to the primary or a replica, is served only after verifying it reflects the latest committed state.

This ensures linearizability but introduces latency because writes block until synchronization completes. Failures or network partitions can temporarily reduce availability until consensus is restored.

Complete Implementation Structure and Code Example

Consider a simplified Python simulation of a strongly consistent key-value store using threading locks and a central coordinator to mimic quorum behavior. This demonstrates the core structure used in production systems like Google Spanner or etcd.

import threading
import time
from typing import Dict, Optional

class StrongConsistencyStore:
    def __init__(self, num_replicas: int = 3):
        self.data: Dict[str, str] = {}          # Shared data store
        self.versions: Dict[str, int] = {}      # Version tracking for linearizability
        self.lock = threading.Lock()            # Global lock simulating quorum coordination
        self.replicas = num_replicas
        self.quorum = (self.replicas // 2) + 1  # Majority quorum

    def write(self, key: str, value: str) -> bool:
        with self.lock:  # Simulate synchronous quorum acknowledgment
            current_version = self.versions.get(key, 0)
            self.data[key] = value
            self.versions[key] = current_version + 1
            print(f"Write committed: {key} = {value} (version {self.versions[key]})")
            # In real systems, this would wait for quorum replicas to acknowledge
            time.sleep(0.05)  # Simulate network round-trip latency
            return True

    def read(self, key: str) -> Optional[str]:
        with self.lock:  # All reads go through coordinated check
            if key in self.data:
                value = self.data[key]
                version = self.versions[key]
                print(f"Read returned: {key} = {value} (version {version})")
                return value
            return None

# Usage demonstration
if __name__ == "__main__":
    store = StrongConsistencyStore()
    store.write("account_balance", "1500.00")
    print("Client A writes balance")
    # Immediate read from any "replica" (simulated)
    print("Client B reads balance:", store.read("account_balance"))
    store.write("account_balance", "1400.00")
    print("Client C reads updated balance:", store.read("account_balance"))

This code enforces strong consistency by serializing all operations under a single lock, ensuring every read sees the latest write. In a real distributed deployment, the lock would be replaced by a consensus algorithm that requires quorum acknowledgments from live replicas. The version counter prevents stale reads even if network delays occur.

Use Cases for Strong Consistency

Strong Consistency is essential in domains where correctness cannot be compromised:

Financial systems and banking applications where account balances must reflect every transaction instantly to prevent double-spending.
Inventory management in e-commerce to ensure stock counts are accurate across all regional warehouses.
Reservation systems such as airline seat booking or hotel room allocation.

Trade-offs of Strong Consistency

While it provides perfect data accuracy, Strong Consistency sacrifices availability during partitions (per the CAP theorem) and increases latency due to synchronization overhead. Systems may return errors or block operations rather than serve potentially inconsistent data.

Eventual Consistency

Eventual Consistency relaxes the immediate guarantee. It promises that if no new writes occur to a data item, all replicas will eventually converge to the same latest value after some unspecified but finite time. Temporary divergence is allowed, and reads may return stale data during the propagation window.

This pattern relies on asynchronous replication. Writes are acknowledged quickly after being applied to a single node or a minimal set, then propagated in the background through mechanisms like gossip protocols, anti-entropy, read repair, or hinting handoff.

How Eventual Consistency Works in Practice

A write succeeds as soon as it reaches one or more nodes (often a single node for maximum availability). Background processes then push the update to other replicas. Conflict resolution strategies—such as last-write-wins (LWW) based on timestamps or vector clocks—resolve any concurrent updates when replicas reconcile.

Clients may see different values briefly, but the system self-heals without manual intervention.

Complete Implementation Structure and Code Example

Below is a full Python implementation simulating an eventually consistent store using asynchronous queues and background reconciliation threads. This mirrors the architecture of Amazon DynamoDB (default mode) or Apache Cassandra.

import threading
import time
import queue
from typing import Dict, Optional

class EventualConsistencyStore:
    def __init__(self, num_replicas: int = 3):
        self.replicas: Dict[int, Dict[str, str]] = {i: {} for i in range(num_replicas)}
        self.versions: Dict[int, Dict[str, int]] = {i: {} for i in range(num_replicas)}
        self.write_queue = queue.Queue()
        self.replica_count = num_replicas
        self.lock = threading.Lock()
        # Start background propagation thread
        self.propagator = threading.Thread(target=self._propagate_updates, daemon=True)
        self.propagator.start()

    def write(self, key: str, value: str, replica_id: int = 0) -> bool:
        with self.lock:
            current_version = self.versions[replica_id].get(key, 0)
            self.replicas[replica_id][key] = value
            self.versions[replica_id][key] = current_version + 1
            self.write_queue.put((key, value, self.versions[replica_id][key]))
            print(f"Write acknowledged on replica {replica_id}: {key} = {value}")
            return True

    def read(self, key: str, replica_id: int = 0) -> Optional[str]:
        value = self.replicas[replica_id].get(key)
        version = self.versions[replica_id].get(key, 0)
        print(f"Read from replica {replica_id}: {key} = {value} (version {version})")
        return value

    def _propagate_updates(self):
        while True:
            try:
                key, value, version = self.write_queue.get(timeout=1)
                with self.lock:
                    for rid in range(self.replica_count):
                        current_version = self.versions[rid].get(key, 0)
                        if version > current_version:
                            self.replicas[rid][key] = value
                            self.versions[rid][key] = version
                time.sleep(0.2)  # Simulate network propagation delay
            except queue.Empty:
                continue

# Usage demonstration
if __name__ == "__main__":
    store = EventualConsistencyStore()
    store.write("user_post", "Hello world", replica_id=0)
    print("Client A reads immediately (may be stale):", store.read("user_post", replica_id=1))
    time.sleep(1)  # Allow propagation
    print("Client B reads after convergence:", store.read("user_post", replica_id=1))

The background thread ensures eventual convergence. In production, vector clocks or CRDTs would replace simple version numbers for more sophisticated conflict handling.

Use Cases for Eventual Consistency

Eventual Consistency excels in high-scale, high-availability scenarios:

Social media feeds and like counters where slight delays in visibility are acceptable.
Content delivery networks (CDN) and caching layers.
DNS systems and email propagation.

Weak Consistency

Weak Consistency provides the fewest guarantees. After a write, subsequent reads may or may not see the update, and there is no assurance that replicas will ever converge. Divergence can persist indefinitely unless explicitly resolved.

This model prioritizes raw performance and availability above all else. Updates are fire-and-forget, with no automatic synchronization or conflict resolution built into the core protocol.

How Weak Consistency Works in Practice

Writes are applied locally to whichever node receives them. Reads return whatever local state exists at that moment. Reconciliation, if any, happens only through application-level logic or manual intervention. There are no background propagators or quorum requirements.

Complete Implementation Structure and Code Example

Here is a complete Python simulation of a weakly consistent store with zero synchronization:

import threading
from typing import Dict, Optional

class WeakConsistencyStore:
    def __init__(self, num_replicas: int = 3):
        self.replicas: Dict[int, Dict[str, str]] = {i: {} for i in range(num_replicas)}
        self.lock = threading.Lock()  # Only for internal safety, not for consistency

    def write(self, key: str, value: str, replica_id: int = 0) -> bool:
        with self.lock:
            self.replicas[replica_id][key] = value
            print(f"Write applied only to replica {replica_id}: {key} = {value}")
            return True

    def read(self, key: str, replica_id: int = 0) -> Optional[str]:
        value = self.replicas[replica_id].get(key)
        print(f"Read from replica {replica_id}: {key} = {value} (no convergence guarantee)")
        return value

# Usage demonstration
if __name__ == "__main__":
    store = WeakConsistencyStore()
    store.write("game_score", "150", replica_id=0)
    print("Client reads from replica 1:", store.read("game_score", replica_id=1))  # Likely None
    store.write("game_score", "200", replica_id=2)
    print("Client reads from replica 0:", store.read("game_score", replica_id=0))  # Still 150

No propagation occurs. Each replica remains isolated unless the application adds custom logic.

Use Cases for Weak Consistency

Weak Consistency suits applications where freshness is secondary to speed:

Multiplayer game leaderboards where occasional staleness does not affect gameplay.
Non-critical caching layers or analytics dashboards.
Highly partitioned sensor networks that tolerate data loss.

Comparing the Patterns

Strong Consistency delivers immediate correctness at the cost of higher latency and reduced availability during failures. Eventual Consistency trades immediate accuracy for superior scalability and availability, relying on time for convergence. Weak Consistency maximizes performance by removing all guarantees, making it suitable only when temporary or permanent divergence is tolerable.

Each pattern aligns with specific positions on the CAP spectrum: Strong Consistency typically favors CP systems, while Eventual and Weak Consistency enable AP designs that remain responsive even under partitions.

The choice depends entirely on the application's tolerance for stale data, required throughput, and acceptable failure modes.

System Design Handbook

Master the complete System Design interview process with real-world architectures, deep-dive explanations, and battle-tested patterns used at top tech companies.

Buy it now at: https://codewithdhanian.gumroad.com/l/ntmcf

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

SQL vs. NoSQL in System Design

CodeWithDhanian — Thu, 02 Apr 2026 19:51:30 +0000

Foundations of SQL Databases

SQL databases, also known as relational databases, organize data into structured tables with predefined schemas. Each table consists of rows and columns, where columns enforce specific data types and constraints. Relationships between tables are established through foreign keys, enabling complex joins to retrieve interconnected data efficiently. This model follows the principles of normalization to minimize data redundancy and ensure data integrity.

The core strength of SQL lies in its adherence to ACID properties: Atomicity, Consistency, Isolation, and Durability. These guarantees make SQL ideal for applications requiring strict transactional consistency, such as financial systems or e-commerce platforms where partial failures cannot occur.

A complete SQL schema for an e-commerce system illustrates this structure. Consider the following full code snippet using PostgreSQL syntax:

-- Complete SQL schema for e-commerce system
CREATE TABLE users (
    user_id SERIAL PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL,
    password_hash TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE products (
    product_id SERIAL PRIMARY KEY,
    name VARCHAR(200) NOT NULL,
    description TEXT,
    price DECIMAL(10,2) NOT NULL,
    stock_quantity INTEGER DEFAULT 0,
    category_id INTEGER REFERENCES categories(category_id)
);

CREATE TABLE categories (
    category_id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL
);

CREATE TABLE orders (
    order_id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(user_id) ON DELETE CASCADE,
    order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    total_amount DECIMAL(10,2) NOT NULL,
    status VARCHAR(20) DEFAULT 'pending'
);

CREATE TABLE order_items (
    order_item_id SERIAL PRIMARY KEY,
    order_id INTEGER REFERENCES orders(order_id) ON DELETE CASCADE,
    product_id INTEGER REFERENCES products(product_id),
    quantity INTEGER NOT NULL,
    price_at_purchase DECIMAL(10,2) NOT NULL
);

-- Example transaction ensuring ACID compliance
BEGIN;
INSERT INTO orders (user_id, total_amount, status) 
VALUES (1, 99.99, 'pending');
INSERT INTO order_items (order_id, product_id, quantity, price_at_purchase) 
VALUES (currval('orders_order_id_seq'), 5, 2, 49.99);
UPDATE products SET stock_quantity = stock_quantity - 2 WHERE product_id = 5;
COMMIT;

This structure enforces referential integrity through foreign keys and cascading deletes. The transaction block guarantees that either all operations succeed or none do, maintaining consistency even under concurrent access. In system design, such SQL setups are typically deployed with master-slave replication for read scalability and vertical scaling by upgrading hardware on a single server.

Foundations of NoSQL Databases

NoSQL databases, or non-relational databases, reject the rigid table structure in favor of flexible data models. They store data in formats such as documents, key-value pairs, wide-column stores, or graphs. Schemas are often dynamic or schema-less, allowing applications to evolve without downtime for migrations. This flexibility prioritizes scalability and performance over strict consistency.

NoSQL systems typically follow BASE principles: Basically Available, Soft state, and Eventual consistency. They excel in horizontal scaling across distributed clusters, making them suitable for high-throughput applications like social media feeds, real-time analytics, or content management systems.

NoSQL encompasses four primary types:

Document stores (e.g., MongoDB) store self-contained JSON-like documents.
Key-value stores (e.g., Redis) provide ultra-fast lookups.
Column-family stores (e.g., Cassandra) handle massive sparse data.
Graph databases (e.g., Neo4j) optimize for relationship-heavy queries.

A complete NoSQL implementation example uses MongoDB for the same e-commerce scenario. The following full code snippet demonstrates document-based storage and operations:

// Complete MongoDB setup and operations for e-commerce system
const { MongoClient } = require('mongodb');
const uri = "mongodb://localhost:27017";
const client = new MongoClient(uri);

async function run() {
  try {
    await client.connect();
    const database = client.db("ecommerce");

    // Collections are created implicitly on first insert
    const usersCollection = database.collection("users");
    const productsCollection = database.collection("products");
    const ordersCollection = database.collection("orders");

    // Insert a user document (schema-less)
    const newUser = {
      username: "johndoe",
      email: "john@example.com",
      passwordHash: "hashedpassword123",
      createdAt: new Date(),
      addresses: [  // Embedded array for flexibility
        { street: "123 Main St", city: "Nairobi", country: "Kenya" }
      ]
    };
    const userResult = await usersCollection.insertOne(newUser);

    // Insert product with dynamic fields
    const newProduct = {
      name: "Wireless Headphones",
      description: "Noise-cancelling over-ear",
      price: 99.99,
      stockQuantity: 50,
      category: "Electronics",
      tags: ["audio", "wireless"]  // Flexible array
    };
    await productsCollection.insertOne(newProduct);

    // Complete order as a single document with embedded items (denormalized for speed)
    const newOrder = {
      userId: userResult.insertedId,
      orderDate: new Date(),
      totalAmount: 99.99,
      status: "pending",
      items: [
        {
          productId: "productObjectIdHere",
          name: "Wireless Headphones",
          quantity: 1,
          priceAtPurchase: 99.99
        }
      ],
      // Embedded shipping info without separate table
      shippingAddress: {
        street: "123 Main St",
        city: "Nairobi",
        country: "Kenya"
      }
    };
    await ordersCollection.insertOne(newOrder);

    // Query example with aggregation for analytics
    const analytics = await ordersCollection.aggregate([
      { $match: { status: "completed" } },
      { $group: { _id: null, totalRevenue: { $sum: "$totalAmount" } } }
    ]).toArray();

    console.log("Analytics:", analytics);
  } finally {
    await client.close();
  }
}
run().catch(console.dir);

This code demonstrates denormalization by embedding related data directly into documents, eliminating the need for joins. Operations occur atomically within a single document, and the system scales horizontally by adding more nodes to the replica set or sharded cluster. In system design, MongoDB would use consistent hashing for data partitioning and replication for high availability.

Comparative Analysis in System Design

SQL and NoSQL differ fundamentally in how they handle data, scale, and guarantee consistency, directly impacting architectural decisions.

Aspect	SQL Databases	NoSQL Databases
Data Model	Relational tables with fixed schema	Flexible documents, key-value, etc.
Schema	Rigid and enforced	Dynamic or schema-less
Query Language	Standardized SQL with joins	Database-specific (e.g., MongoDB Query Language)
Consistency	ACID (strong guarantees)	BASE (eventual consistency)
Scalability	Vertical scaling preferred	Horizontal scaling across clusters
Use Case Fit	Transactions, complex relationships	High volume, unstructured data, real-time

SQL shines when data integrity and complex queries are paramount. NoSQL excels when velocity, variety, and volume dominate, as in big data pipelines or global user bases.

In a distributed system design, a hybrid approach often emerges. User authentication and financial transactions might reside in PostgreSQL for ACID compliance, while user activity logs and product recommendations use Cassandra or MongoDB for horizontal scaling and eventual consistency. Data partitioning via sharding in NoSQL contrasts with partitioning strategies in SQL that rely more on read replicas.

Practical Implementation in Distributed Systems

System designers must evaluate trade-offs when selecting databases. For a high-traffic social platform, SQL might handle user profiles with strong consistency via two-phase commits in rare cross-service transactions. NoSQL would manage feeds using event-driven architecture with message queues publishing changes for eventual propagation.

Leader election and consensus algorithms like Raft ensure NoSQL clusters remain available during node failures. SQL clusters often employ multi-master replication with careful conflict resolution.

The choice influences microservices architecture: each service owns its database, enforcing database-per-service patterns. API gateways route requests, while circuit breakers and retry mechanisms handle transient failures across database boundaries.

This comprehensive understanding equips system designers to architect resilient, scalable solutions tailored to specific requirements.

To help visualize the concepts discussed, here is one complete image:

System Design Handbook

To master these concepts and many more, purchase the complete System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf.

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

PACELC Theorem in System Design

CodeWithDhanian — Thu, 02 Apr 2026 19:33:47 +0000

The PACELC Theorem represents a foundational advancement in understanding the inherent trade-offs that define modern distributed systems. Developed as a direct extension of the CAP Theorem, it provides architects and engineers with a more complete framework for reasoning about system behavior under both failure conditions and normal operations. Where earlier models focused narrowly on rare network failures, the PACELC Theorem acknowledges that consistency, availability, and latency constantly interact in real production environments.

The Evolution from CAP to PACELC

The CAP Theorem established that in the presence of a network partition, a distributed system can guarantee only two out of three properties: Consistency, Availability, and Partition Tolerance. This insight proved invaluable for designing fault-tolerant architectures. However, it left a critical gap unaddressed. The CAP Theorem offered no guidance on system behavior during the vast majority of time when no network partition exists. In practice, distributed databases and microservices spend most of their operational life in a healthy state, yet they still face unavoidable trade-offs.

The PACELC Theorem, proposed by Daniel Abadi, bridges this exact limitation. It formalizes the reality that even without failures, designers must choose between latency and consistency. The theorem therefore expands the conversation from failure-only scenarios to the continuous operational reality of distributed systems.

Decoding the PACELC Acronym

The PACELC Theorem breaks down into two distinct decision points that every replicated distributed system must navigate.

P stands for Partition. When a network partition occurs, nodes or groups of nodes become unable to communicate. At this moment the system must decide between A (Availability) and C (Consistency).

A represents Availability: the guarantee that every request receives a non-error response, even if the response reflects stale data.

C represents Consistency: the guarantee that all nodes return the most recent successful write, ensuring linearizability across the system.

E stands for Else. This clause addresses the normal operating state when no network partition is present and the cluster functions with full connectivity.

In the Else case, the system must still choose between L (Latency) and C (Consistency).

L represents Latency: the time taken to complete read or write operations. Lower latency improves user experience and throughput but often requires relaxing guarantees about data freshness.

C again represents Consistency, now enforced through synchronous coordination that inevitably increases response times.

The complete formulation therefore states: in the case of a network partition (P), a distributed system can trade off availability (A) and consistency (C); else (E), when the system operates normally, it must trade off latency (L) and consistency (C).

The Partition Scenario in Depth

During a network partition, the system faces an existential choice. Prioritizing Availability means continuing to serve requests from whichever partition can respond. Some nodes may return stale data, but the service remains usable. Prioritizing Consistency means refusing requests that cannot be verified against the latest state, potentially rendering parts of the system unavailable until the partition heals.

This decision directly maps to the CAP Theorem but gains precision when combined with the Else clause. Real systems rarely stay partitioned indefinitely, so the PACELC Theorem forces designers to consider both the failure mode and the recovery behavior.

The Normal Operation Scenario

The true power of the PACELC Theorem emerges in the Else case. Even with perfect connectivity, synchronous replication across multiple nodes introduces latency. A write must reach a quorum or all replicas before acknowledgment, increasing response time. Asynchronous replication reduces latency dramatically but risks temporary inconsistency until replication catches up.

This latency versus consistency trade-off occurs constantly. High-traffic applications serving millions of users per second cannot afford the overhead of strong consistency on every operation. Conversely, financial or inventory systems cannot tolerate even brief windows of stale data.

System Classifications under PACELC

Distributed systems fall into four primary categories based on their PACELC choices:

PA/EL systems prioritize Availability during partitions and Latency during normal operation. They favor eventual consistency models.
PA/EC systems prioritize Availability during partitions but enforce Consistency during normal operation.
PC/EC systems prioritize Consistency in both scenarios, accepting potential unavailability and higher latency.
PC/EL configurations remain rare because sacrificing Availability during partitions while accepting Latency trade-offs in normal operation offers limited practical benefit.

Real-World Database Implementations

Apache Cassandra operates as a classic PA/EL system. It uses a tunable consistency model allowing developers to choose consistency levels per query, but defaults to behaviors that favor Availability and low latency. During a partition, Cassandra continues serving requests from available nodes. Under normal conditions, writes return quickly after reaching a single node or local quorum, with background repair mechanisms ensuring eventual consistency.

Amazon DynamoDB follows the same PA/EL pattern. It delivers single-digit millisecond responses at global scale by defaulting to eventual consistency. Developers can request strongly consistent reads when needed, but this option explicitly increases latency and reduces Availability under certain failure modes, demonstrating the PACELC trade-off in action.

MongoDB typically behaves as a PA/EC system. It can maintain Availability during partitions while guaranteeing Consistency for reads and writes under normal operation through primary-secondary replication and careful write concern settings.

Google Spanner and HBase exemplify PC/EC systems. They refuse to compromise Consistency even during partitions, using sophisticated consensus protocols like Paxos or Raft. Writes may block or fail until quorum agreement, and reads always reflect the latest committed state. The resulting higher latency and occasional unavailability represent the deliberate cost of absolute Consistency.

Practical Code Examples

To illustrate these concepts concretely, consider the following complete examples that demonstrate PACELC trade-offs in practice.

Example 1: Tunable Consistency in Cassandra

The following CQL statements show how Cassandra exposes the latency-consistency choice directly to the application layer:

-- Table creation for a user profile store
CREATE TABLE user_profiles (
    user_id UUID PRIMARY KEY,
    name TEXT,
    email TEXT,
    last_updated TIMESTAMP
) WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': 3};

-- Write with low latency (PA/EL default behavior)
INSERT INTO user_profiles (user_id, name, email, last_updated)
VALUES (uuid(), 'Alice Johnson', 'alice@example.com', now())
USING CONSISTENCY ONE;

-- Write with strong consistency (trading latency for C)
INSERT INTO user_profiles (user_id, name, email, last_updated)
VALUES (uuid(), 'Alice Johnson', 'alice@example.com', now())
USING CONSISTENCY QUORUM;

-- Read with low latency (may return stale data)
SELECT * FROM user_profiles WHERE user_id = 123e4567-e89b-12d3-a456-426614174000
USING CONSISTENCY ONE;

-- Read with strong consistency (higher latency, guaranteed latest data)
SELECT * FROM user_profiles WHERE user_id = 123e4567-e89b-12d3-a456-426614174000
USING CONSISTENCY QUORUM;

In the ONE consistency level, the operation completes after a single replica acknowledges the write or read, delivering minimal latency at the cost of possible temporary inconsistency. The QUORUM level requires majority acknowledgment, enforcing stronger Consistency while increasing latency and reducing effective Availability during partial failures.

Example 2: Python Simulation of Latency versus Consistency Trade-off

The following complete Python implementation demonstrates a simplified replicated key-value store that lets the developer choose between low-latency asynchronous replication and high-consistency synchronous replication:

import threading
import time
import uuid
from typing import Dict, Optional

class ReplicaNode:
    def __init__(self, node_id: str):
        self.node_id = node_id
        self.store: Dict[str, str] = {}
        self.version: Dict[str, int] = {}

    def write(self, key: str, value: str, version: int):
        self.store[key] = value
        self.version[key] = version

class DistributedKVStore:
    def __init__(self, num_replicas: int = 3):
        self.replicas = [ReplicaNode(f"node-{i}") for i in range(num_replicas)]
        self.global_version = 0
        self.lock = threading.Lock()

    def write_elastic(self, key: str, value: str) -> str:
        """PA/EL style: low latency, eventual consistency"""
        self.global_version += 1
        version = self.global_version

        # Write to primary immediately
        self.replicas[0].write(key, value, version)

        # Asynchronous replication to other replicas (low latency)
        for replica in self.replicas[1:]:
            threading.Thread(
                target=replica.write,
                args=(key, value, version),
                daemon=True
            ).start()

        return f"Written with version {version} (eventual consistency)"

    def write_strong(self, key: str, value: str) -> str:
        """PC/EC style: higher latency, strong consistency"""
        self.global_version += 1
        version = self.global_version

        # Synchronous replication to ALL replicas
        threads = []
        for replica in self.replicas:
            t = threading.Thread(
                target=replica.write,
                args=(key, value, version)
            )
            t.start()
            threads.append(t)

        # Wait for all replicas (higher latency)
        for t in threads:
            t.join()

        return f"Written synchronously with version {version} (strong consistency)"

    def read(self, key: str, strong: bool = False) -> Optional[str]:
        if strong:
            # Read from primary after ensuring propagation
            time.sleep(0.05)  # Simulate synchronization delay
            return self.replicas[0].store.get(key)
        else:
            # Return from any replica (low latency, possible staleness)
            for replica in self.replicas:
                if key in replica.store:
                    return replica.store[key]
        return None

# Usage demonstration
store = DistributedKVStore()

print(store.write_elastic("user:123", "Alice"))  # Fast, eventual
print(store.write_strong("user:456", "Bob"))     # Slower, strong

time.sleep(0.1)  # Allow async replication to complete
print("Strong read:", store.read("user:123", strong=True))
print("Elastic read:", store.read("user:123", strong=False))

This simulation makes the PACELC trade-off explicit. The write_elastic method returns almost instantly while background threads handle replication, embodying PA/EL behavior. The write_strong method blocks until every replica acknowledges, providing PC/EC guarantees at the measurable cost of increased latency.

Designing Systems with PACELC in Mind

When architecting new distributed systems, evaluate requirements against both branches of the PACELC Theorem. High-throughput applications such as social feeds, recommendation engines, or IoT telemetry streams benefit from PA/EL designs. Mission-critical systems handling financial transactions, inventory management, or medical records demand PC/EC approaches despite the performance penalty.

Hybrid strategies also exist. Many production systems implement dynamic consistency tuning based on context. A single API endpoint may offer both eventual and strong read paths, allowing clients to select the appropriate latency-consistency balance per request.

The PACELC Theorem ultimately equips system designers with the vocabulary and mental model necessary to make intentional, evidence-based decisions rather than defaulting to marketing claims or oversimplified diagrams.

System Design Handbook

If you found this deep dive valuable, grab the complete System Design Handbook packed with 40+ essential concepts, real architectures, and production-grade examples: https://codewithdhanian.gumroad.com/l/ntmcf

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

CAP Theorem in System Design

CodeWithDhanian — Wed, 01 Apr 2026 10:43:08 +0000

The Core Principle Behind Distributed Data Systems

The CAP Theorem stands as one of the foundational concepts in modern system design. It defines the fundamental limitations that every distributed system must confront when handling data across multiple nodes connected over a network. At its heart, the CAP Theorem asserts that in any distributed system, it is impossible to simultaneously guarantee all three of the following properties: Consistency, Availability, and Partition Tolerance. Designers must therefore choose only two out of these three properties, accepting the inevitable trade-offs that arise from the choice.

This theorem emerged from the practical realities of building large-scale, fault-tolerant systems where nodes communicate over unreliable networks. In such environments, failures are not exceptions but expected occurrences. The CAP Theorem forces architects to make deliberate decisions about what matters most for their specific use case, whether it is financial integrity, user experience, or system resilience.

Understanding the Three Properties in Depth

To fully grasp the CAP Theorem, each property must be examined with precision.

Consistency means that every read operation receives the most recent write or an error. In a consistent system, all nodes reflect the same data at the same time. If one node updates a value, every subsequent read from any node returns the updated value immediately. This property ensures that users and applications always see a single, truthful view of the data, eliminating discrepancies that could lead to incorrect business decisions or user confusion.

Availability guarantees that every request sent to the system receives a non-error response, regardless of the state of individual nodes. An available system never refuses a request; it always returns data, even if that data comes from a potentially stale copy. This property prioritizes responsiveness and ensures that the system remains operational for all users at all times.

Partition Tolerance requires the system to continue operating despite network partitions, which occur when communication between subsets of nodes is lost or severely delayed. A partition-tolerant system maintains functionality even when some nodes cannot reach others. This property acknowledges the harsh reality of distributed environments where network issues, hardware failures, or geographic distances inevitably create temporary disconnections.

The CAP Theorem proves that during a network partition, a system can satisfy at most two of these properties. If Consistency and Availability are both required, the system cannot tolerate partitions, which is unrealistic for any truly distributed architecture.

Why Partition Tolerance Cannot Be Ignored

Network partitions are inevitable in distributed systems. Nodes may reside in different data centers, regions, or even continents. Cables can be cut, routers can fail, and traffic spikes can overwhelm connections. Because of this reality, Partition Tolerance becomes a non-negotiable requirement for any production-grade distributed system. Architects must therefore design around the choice between Consistency and Availability while always preserving Partition Tolerance.

When a partition occurs, the system faces a stark decision. It can either:

Maintain Consistency by refusing writes or reads on one side of the partition until synchronization is restored, thereby sacrificing Availability.
Maintain Availability by allowing both sides to accept reads and writes independently, thereby sacrificing Consistency until the partition heals and reconciliation occurs.

This choice defines the system’s behavior under failure and directly influences its architecture, data model, and operational procedures.

The Practical Trade-offs: CP, AP, and the Myth of CA

Systems are classified based on their prioritized properties during partitions.

CP systems (Consistency and Partition Tolerance) prioritize Consistency above all else. During a network partition, these systems may block certain operations on one side to prevent divergent data. Reads and writes are only permitted when the system can guarantee the latest state. This approach ensures absolute data accuracy but may result in temporary unavailability for some users. Banking systems and inventory management platforms often adopt CP designs because even momentary inconsistencies could lead to overdrafts or overselling.

AP systems (Availability and Partition Tolerance) prioritize Availability above all else. During a partition, every node continues to serve requests using whatever data is locally available. This leads to eventual reconciliation once the partition resolves, but users may temporarily see stale or conflicting information. Social media feeds, recommendation engines, and content delivery platforms frequently choose AP designs to ensure users never encounter downtime or error messages.

CA systems (Consistency and Availability) would theoretically provide both Consistency and Availability but only if no partitions ever occur. In practice, true CA systems exist only in single-node or tightly coupled environments where network partitions are impossible. Because real-world distributed systems always face partitions, CA is not a viable classification for scalable architectures. The CAP Theorem therefore reduces the meaningful choices to CP or AP.

Real-World Implications in Database Design

Different databases embody these trade-offs explicitly in their configurations.

A CP database might use quorum-based reads and writes, requiring a majority of nodes to agree before committing or responding. If a partition splits the cluster such that no quorum is possible on one side, that side will reject requests to preserve Consistency. This ensures that every successful read reflects the absolute latest write, but it may return errors or timeouts during partitions.

An AP database, by contrast, allows any node to accept writes and serve reads immediately. It employs techniques such as hinted handoffs, read repairs, and anti-entropy mechanisms to propagate updates once connectivity returns. Users experience seamless operation, but the system must later resolve conflicts through last-write-wins, vector clocks, or custom merge logic.

Hybrid approaches exist where applications tune behavior at the query level, requesting strong Consistency for critical operations while accepting eventual Consistency for others. This fine-grained control allows systems to balance the CAP Theorem trade-offs dynamically based on business requirements.

Simulating the CAP Trade-off with Code

To illustrate the practical differences, consider a simplified distributed key-value store implemented in Python. The following complete code snippet simulates three nodes and demonstrates both CP and AP behaviors during a simulated network partition.

import threading
import time
from typing import Dict, Optional

class Node:
    def __init__(self, node_id: str):
        self.node_id = node_id
        self.data: Dict[str, str] = {}
        self.is_partitioned = False
        self.lock = threading.Lock()

    def write(self, key: str, value: str, require_quorum: bool = False) -> bool:
        if self.is_partitioned and require_quorum:
            print(f"Node {self.node_id}: Write blocked - partition detected (CP behavior)")
            return False
        with self.lock:
            self.data[key] = value
            print(f"Node {self.node_id}: Wrote {key} = {value}")
            return True

    def read(self, key: str, require_quorum: bool = False) -> Optional[str]:
        if self.is_partitioned and require_quorum:
            print(f"Node {self.node_id}: Read blocked - partition detected (CP behavior)")
            return None
        with self.lock:
            value = self.data.get(key)
            print(f"Node {self.node_id}: Read {key} = {value}")
            return value

def simulate_partition(nodes: list[Node], partition_duration: float = 5.0):
    print("Simulating network partition...")
    for node in nodes:
        node.is_partitioned = True
    time.sleep(partition_duration)
    for node in nodes:
        node.is_partitioned = False
    print("Partition healed - reconciliation would occur here")

# CP System Simulation (Consistency + Partition Tolerance)
print("=== CP System Simulation ===")
cp_nodes = [Node("CP-Node1"), Node("CP-Node2"), Node("CP-Node3")]

# Initial write across all nodes (simulating replication)
for node in cp_nodes:
    node.write("user_balance", "1000", require_quorum=False)

# Simulate partition on Node3
cp_nodes[2].is_partitioned = True
cp_nodes[0].write("user_balance", "1200", require_quorum=True)  # Succeeds on majority side
cp_nodes[2].read("user_balance", require_quorum=True)           # Blocked - preserves consistency

# Heal partition
cp_nodes[2].is_partitioned = False
print("Reconciliation complete - all nodes now see 1200\n")

# AP System Simulation (Availability + Partition Tolerance)
print("=== AP System Simulation ===")
ap_nodes = [Node("AP-Node1"), Node("AP-Node2"), Node("AP-Node3")]

for node in ap_nodes:
    node.write("user_balance", "1000", require_quorum=False)

# Simulate partition on Node3
ap_nodes[2].is_partitioned = True
ap_nodes[0].write("user_balance", "1200", require_quorum=False)  # Succeeds anyway
ap_nodes[2].write("user_balance", "900", require_quorum=False)   # Succeeds anyway (temporary inconsistency)
ap_nodes[2].read("user_balance", require_quorum=False)           # Returns local value immediately

# Heal partition
ap_nodes[2].is_partitioned = False
print("Reconciliation would merge conflicting values using last-write-wins or custom logic")

In the CP simulation, the partitioned node blocks operations to guarantee Consistency. In the AP simulation, all nodes remain responsive, accepting the risk of temporary divergence that must be resolved later. This code structure mirrors the architectural decisions made by real distributed databases, showing how the CAP Theorem directly influences implementation details such as locking strategies, quorum requirements, and conflict resolution mechanisms.

Designing Systems That Embrace the CAP Theorem

Mastering the CAP Theorem requires more than theoretical knowledge; it demands that every component of the architecture reflects the chosen trade-offs. Replication strategies, consensus protocols, and client-side retry logic must all align with whether the system prioritizes Consistency or Availability. Monitoring tools track partition events, while operational runbooks define how to handle reconciliation when connectivity returns. By internalizing these principles, system designers create robust, predictable distributed systems capable of scaling to millions of users while respecting the immutable laws of distributed computing.

System Design Handbook

If you found this deep dive into the CAP Theorem valuable and want to master all 40 essential concepts with complete code examples, architectures, and real-world implementations, purchase the full System Design Handbook here: https://codewithdhanian.gumroad.com/l/ntmcf

Buy me coffee to support my content at: https://ko-fi.com/codewithdhanian

Database Replication (Master-Slave, Multi-Master) in System Design

CodeWithDhanian — Wed, 01 Apr 2026 04:12:50 +0000

Database replication forms a cornerstone of robust system design, enabling databases to achieve higher scalability, availability, and data durability in large-scale applications. By duplicating data across multiple nodes, replication ensures that read and write operations can be distributed intelligently while protecting against single points of failure. This section explores the two dominant replication architectures—Master-Slave Replication and Multi-Master Replication—with complete technical depth, configuration examples, and operational realities.

Introduction to Database Replication

Database replication is the process of copying and maintaining database objects and data changes across multiple database instances. In a distributed system, a primary database records all modifications in a transaction log, which is then transmitted to one or more secondary instances. These secondary instances replay the changes, keeping their data sets synchronized.

The core objectives of replication are threefold. First, it delivers horizontal scalability by allowing read-heavy workloads to be offloaded to additional nodes. Second, it provides high availability through automatic failover mechanisms when a node becomes unreachable. Third, it improves disaster recovery by maintaining geographically dispersed copies that survive hardware failures, network partitions, or data center outages.

Replication operates at different levels: statement-based, row-based, or mixed. Statement-based replication logs the exact SQL statements executed on the source, while row-based replication logs the actual before-and-after row images, offering greater precision during schema changes or non-deterministic operations. Modern database engines default to row-based for its reliability.

Master-Slave Replication

Master-Slave Replication, also known as primary-replica replication, designates one node as the master responsible for all write operations while one or more slave nodes handle read operations and serve as hot standbys. This architecture is the most widely adopted pattern for read scaling because it maintains strong write consistency on a single master.

Architecture of Master-Slave Replication

The master node accepts all INSERT, UPDATE, and DELETE statements. Every committed transaction is written to the binary log (in MySQL) or WAL (Write-Ahead Log in PostgreSQL). A dedicated replication thread on each slave connects to the master, reads the log events, and applies them locally through an SQL thread or apply process.

Slaves are typically configured in read-only mode to prevent accidental writes. Applications route write queries exclusively to the master and read queries to any available slave. This separation is commonly implemented via connection pooling libraries or ORM routing logic.

Replication Modes: Synchronous and Asynchronous

Asynchronous replication is the default mode. The master commits a transaction, writes it to its binary log, and immediately returns success to the client. The replication to slaves happens in the background. This delivers the lowest write latency but introduces replication lag—a period during which slaves have not yet applied the latest changes. Lag can be monitored using metrics such as Seconds_Behind_Master in MySQL.

Synchronous replication requires the master to wait for acknowledgment from at least one slave before committing. This guarantees zero lag for acknowledged writes but increases write latency and reduces overall throughput. Semi-synchronous replication strikes a balance: the master waits for one slave to receive the event (but not necessarily apply it), providing improved durability with acceptable performance.

Setting Up Master-Slave Replication in MySQL

Below are complete, production-ready configuration files and commands for establishing Master-Slave Replication using MySQL 8.0+.

Master Server Configuration (/etc/mysql/my.cnf):

[mysqld]
server-id               = 1
log-bin                 = mysql-bin
binlog-format           = ROW
expire-logs-days        = 7
innodb_flush_log_at_trx_commit = 1
sync_binlog             = 1
gtid_mode               = ON
enforce_gtid_consistency = ON

Explanation of key parameters:

server-id = 1: Uniquely identifies the master in the replication topology.
log-bin = mysql-bin: Enables binary logging, the foundation of replication.
binlog-format = ROW: Ensures precise, row-level changes are logged.
gtid_mode = ON: Activates Global Transaction Identifiers, simplifying failover and topology management.

Slave Server Configuration (/etc/mysql/my.cnf):

[mysqld]
server-id               = 2
relay-log               = relay-log
read_only               = 1
gtid_mode               = ON
enforce_gtid_consistency = ON

Slave setup commands (executed after starting the slave instance):

-- On Master first (to obtain log position)
SHOW MASTER STATUS\G

-- On Slave
CHANGE MASTER TO
    MASTER_HOST = '192.168.1.100',
    MASTER_USER = 'repl_user',
    MASTER_PASSWORD = 'strong_password',
    MASTER_AUTO_POSITION = 1;

START SLAVE;

-- Verify replication status
SHOW SLAVE STATUS\G

The MASTER_AUTO_POSITION = 1 leverages GTID for automatic positioning, eliminating manual log file and position tracking. The replication user must have the REPLICATION SLAVE privilege granted on the master.

Multi-Master Replication

Multi-Master Replication allows multiple nodes to accept write operations simultaneously. Every master acts as both a write and read endpoint, with changes propagated bidirectionally. This architecture is essential for globally distributed applications requiring low-latency writes from multiple regions.

Architecture of Multi-Master Replication

In a multi-master setup, each node maintains its own binary log and replication threads that push changes to every other node. Circular replication or full-mesh replication topologies are common. Global Transaction Identifiers become critical to detect and prevent infinite loops of replicated events.

Because multiple masters can modify the same row concurrently, conflict resolution is mandatory. Without it, data divergence occurs.

Conflict Resolution Strategies

Last Writer Wins (LWW) uses timestamps or logical clocks to decide which change prevails. Version vectors or CRDTs (Conflict-free Replicated Data Types) provide more sophisticated merging. Custom business logic can also be implemented via triggers or application-level reconciliation jobs.

MySQL Group Replication offers built-in conflict detection using a certification-based mechanism that aborts conflicting transactions.

Setting Up Multi-Master Replication in MySQL (Circular Example)

For a two-node active-active circular multi-master setup:

Node 1 Configuration (/etc/mysql/my.cnf):

[mysqld]
server-id               = 1
log-bin                 = mysql-bin
binlog-format           = ROW
gtid_mode               = ON
enforce_gtid_consistency = ON
auto-increment-increment = 2
auto-increment-offset    = 1

Node 2 Configuration (/etc/mysql/my.cnf):

[mysqld]
server-id               = 2
log-bin                 = mysql-bin
binlog-format           = ROW
gtid_mode               = ON
enforce_gtid_consistency = ON
auto-increment-increment = 2
auto-increment-offset    = 2

Establish bidirectional replication:

On Node 1:

CHANGE MASTER TO
    MASTER_HOST = 'node2-ip',
    MASTER_USER = 'repl_user',
    MASTER_PASSWORD = 'strong_password',
    MASTER_AUTO_POSITION = 1;

START SLAVE;

On Node 2 (pointing back to Node 1):

CHANGE MASTER TO
    MASTER_HOST = 'node1-ip',
    MASTER_USER = 'repl_user',
    MASTER_PASSWORD = 'strong_password',
    MASTER_AUTO_POSITION = 1;

START SLAVE;

The auto-increment settings prevent primary key collisions across masters. In production, MySQL Group Replication or Galera Cluster is preferred over manual circular setups because they provide multi-primary mode with automatic quorum and flow control.

Operational Considerations in System Design

Master-Slave Replication excels in read-heavy applications such as content delivery platforms and analytics dashboards. It simplifies consistency guarantees but creates a write bottleneck on the single master.

Multi-Master Replication supports geo-distributed systems and high write throughput but demands careful conflict handling and monitoring of replication lag across all nodes. Application code must be designed to tolerate brief inconsistencies or implement compensating transactions.

Both strategies integrate with load balancers, connection routers, and orchestration tools like ProxySQL or PgBouncer to direct traffic intelligently. Monitoring of replication lag, binary log size, and thread status is non-negotiable for production stability.

A complete system design decision between Master-Slave and Multi-Master ultimately depends on the workload profile, latency requirements, and tolerance for eventual consistency.

System Design Handbook: For comprehensive coverage of all system design concepts including advanced replication patterns, load balancing, caching, and real-world case studies, purchase the System Design Handbook at https://codewithdhanian.gumroad.com/l/ntmcf. This resource equips engineers and architects with the practical knowledge needed to design scalable, production-grade systems.