Distributed Systems
- A system composed of multiple components running on different computers that communicate over a network to work together coherently.
- “A distributed system is a collection of independent computers that appears to its users as a single coherent system.” - Tanenbaum and van Steen
Consistency in Distributed Systems
- Consistency is a broader term that means multiple things depending on context.
- In database context (ACID): consistency means the system moves from one valid state to another, preserving all constraints and invariants (e.g. unique fields, foreign keys, business rules).
- Read-after-write consistency: what value you see when you read data after writes happen across multiple nodes, read last written data.
- Replication: Replica should be in the same state as other replicas (when? it can vary depending on your context).
- As shown there are multiple definitions for consistency, and many to choose from.
- You can achieve different consistency levels in your application.
Consistency Models
- Consistency models are patterns regarding the different levels of consistency we can achieve in different aspects of a distributed system. For example, replication consistency, read-after-write consistency, and many others. People often informally call them “weak consistency” and “strong consistency”, but there are shades of consistency between these.
- When talking about consistency models and focusing on data replication and read-after-write aspects, consistency is generally observed through reads, but it depends on how writes are handled.
- For read operations: the guarantees about which value is returned by a read in the presence of replication and concurrent writes.
- For write operations: the guarantees about how writes are ordered and propagated across replicas.
- Each consistency model has trade-offs, and understanding them is crucial for selecting the best option for your business context. Today I'll discuss what is called “strong consistency”. I will not refer to it as “strong consistency” because this isn't a formal term. The formal one is linearizability.
Example where Consistency is a Point of Attention
- Imagine a scenario where you have an e-commerce system built as a distributed system, similar to Amazon, containing multiple services.
- Look at the diagram below to see these services:
- Customer A accesses your e-commerce system and buys a product. Then OrderService creates an order, PaymentService processes the payment, and only then StockService updates the stock. But imagine that during the payment process, Customer B also buys the same product, and there are not enough units of this product in stock.
- This is a scenario in which consistency becomes a problem that requires an architectural decision using the most appropriate consistency model for your context.
Business-Driven Consistency Trade-offs
- Given the example above, you could imagine different requirements:
Prevent oversell:
- “The better option is to avoid selling products we don’t have.” For this, you would prefer not to allow a purchase if the product isn’t available in stock.
- You avoid inconsistency by enforcing constraints at write time, typically using atomic operations.
- Atomic: the operation happens entirely or nothing changes. If it fails, no state is modified.
Allow oversell:
- “It is acceptable to let multiple users buy because I can figure out how to restock later, but I cannot lose a sale.” For that, you would allow purchases even if stock is not available.
- You would then need to handle it with business options such as refunds or waiting for stock.
- As you can see, these are different consistency trade-offs. One is more strict, and the other is more relaxed. We cannot say one is better than the other, it depends on the context and business rules.
Trade-off: Availability vs Consistency
- Sometimes you will want to favor high availability over consistency, but there are cases where linearizability is crucial. There is no fixed rule, you need to identify the specific parts of your system and their functional requirements (what the system must do) as well as their non-functional requirements (how the system should behave, such as performance, availability, and consistency constraints).
Linearizability - Multiple replicas behave as a single system
- We have multiple servers and database replicas, but all operations behave as if they were executed on a single database. Every operation takes effect atomically from the user's perspective, not necessarily in the database internals. The focus is on client-observable behavior: when and what the operations return. We ignore the internal replication mechanism.
-
Atomic operations in distributed systems:
- Either all nodes commit, or all abort.
- If any node crashes, the system ensures the operation is aborted or completed safely.
- The concept of linearizability is not restricted to distributed systems. It is also used in concurrent programming in general with shared memory.
- Up-to-date value is returned for subsequent reads after one write. “Up-to-date” means that if write A finishes before read B starts, there is a real-time dependency to respect.
- Image reference: Martin Kleppmann
Linearizability != Serializability
- Linearizability: operations appear to execute in real time order. If A finishes before B starts, then B must see the effect of A.
- Serializability: transactions produce a result equivalent to some serial (one-by-one) execution order. The order does not have to follow real time.
Examples of Linearizability with Real-time Dependency
- Case1: A writes x = 1 → A finishes → B reads x = 1 (even for different DB replicas)
-
Case 2: (this is subject of common confusion - overlapping operations)
- A starts x = 1
- B starts reading x (while A is still in progress) → returns x = 0
- A finishes x = 1
- This is linearizable and valid, because the read overlaps the write, so the system can order it as: B → A
-
Case 3: (overlapping operations the other way is also valid)
- A starts x = 1
- B starts reading x (while A is still in progress)
- A finishes x = 1
- B returns x = 1
Ways of Achieving Linearizability
- Typically requires synchronous data replication across multiple servers OR coordination mechanisms (such as quorum) to ensure necessary replicas reflect the latest write before reads are served.
- Below there is a sync replication example, but it could be async using quorum or leader consensus.
- And below there is a linearizability issue in which the write did not respect quorum, so when reading, client 3 got an invalid value (v0). The write happened from client 1 only to DB node A, so when reading from a quorum of 2 nodes does not work correctly, because B and C are not up to date with real-time dependencies, which are required for a linearizable system.
- Image references: Martin Kleppmann
- To solve the issue above following the quorum pattern, some solutions:
- Solution 1 - write quorum: set applied to a minimum quorum of N nodes - in this case 2 of 3.
- Solution 2 - read repair: still allow set only on one DB node, but once the server (client requesting data) identifies in a get request that a node has stale data, it will resend a set with the newest data to all nodes containing old data and will wait for at least one DB node to acknowledge.
- In our case, client 2 on the get request would identify that node A is up to date and node B is outdated, then would send a set request to both B and C and wait for at least one to respond.
- We will discuss more about practical implementations of consistency models such as quorum in further articles.
Use cases of Linearizability:
- Systems that require strong consistency guarantees where operations must reflect the most recent completed write. Examples: banking/financial systems (you don’t want stale balances when updating accounts), flight booking systems (you don’t want multiple people successfully booking the same seat).
- no stale reads after a completed write
- strong ordering of operations
- correctness under concurrency
Linearizability Pros and Cons:
- Trade-off: serving up-to-date data, but with higher latency and coordination cost. This involves a trade-off between consistency and availability.
-
Pros:
- No stale data for completed writes, reads always see the latest completed value.
- Simpler application logic in some cases because there is no need to handle read conflicts or inconsistent states.
- Guarantees correctness in concurrent systems even when many operations happen at the same time.
-
Cons:
- Higher latency, since writes (and sometimes reads) require coordination across replicas.
- Lower throughput, because coordination limits how many operations can be processed in parallel. Higher latency also contributes to reduced throughput.
- Reduced availability, because operations may block or fail if enough replicas are not reachable.
- More complex infrastructure, since it requires coordination protocols (e.g., leader election, quorum, consensus-based replication).
Linearizability Common Mistakes
- A strongly consistent storage layer can still be part of a system that serves stale data if caches are involved. In that case, the database state is correct, but the overall system may return stale cached values, so it is no longer linearizable from the client’s perspective.
- When a system implements linearizability, the cache must not break the guarantee that reads return the most recent committed write.
Exemplifying:
- Think of a CDN or browser cache. With TTL, users may see stale data even if the database has the latest value. This is a freshness issue, not a consistency issue.
- In distributed systems, even with linearizability, stale reads can still happen outside the database layer due to caches or read paths that bypass the primary source of truth.
- In this case:
- Database: still linearizable
- Entire system (DB + cache + app): NOT linearizable
- If you don't know cache basics, check my last article: https://dev.to/godinhojoao/cache-layers-in-modern-applications-563e
Final thoughts:
- Trade-off between availability and consistency is important depending on your context. Since we can’t guarantee both, if some nodes are down, a linearizability model may become unavailable, but eventual consistency can still work even with some nodes down.
- It is easier to ensure linearizability when your application doesn't handle multiple concurrent writes.
- And probably you've noticed that:
- “As the system grows, despite the technology, you can’t rely on consistency.” - Dino Esposito
- You can build linearizable systems at large scale, but this highlights the cost in latency, coordination, and complexity, not to mention availability trade-offs. If it is required in your case, it is still the right choice.
- Want to understand more about consistency models? I’ll continue developing content in this series about consistency models and later cover practical implementations of each model.
Top comments (0)