DEV Community: Venkatesan Ramar

Distributed Locking in Practice: Guarantees, Failure Scenarios and Better Alternatives (4/4)

Venkatesan Ramar — Wed, 19 Aug 2026 08:05:19 +0000

22. The Best Distributed Lock Is Often No Lock at All

By now, we've explored distributed locks, leases, fencing tokens, leader election, and consensus.

Each of these mechanisms exists to solve a specific coordination problem in distributed systems, where multiple machines must agree on who is allowed to do what and when.

A natural conclusion from studying all of them might be that building reliable distributed systems requires increasingly sophisticated coordination mechanisms.

However, in practice, experienced engineers often arrive at the opposite conclusion after working with these systems at scale. Before introducing any form of distributed coordination, they tend to step back and ask a more fundamental question:

Can we re-design the system so that coordination isn't necessary in the first place?

This question is important because every coordination mechanism introduces real cost into the system.

It adds additional network communication between services. It increases the number of failure scenarios that must be handled correctly. It introduces operational complexity that must be monitored, debugged, and maintained. It often increases latency, since coordination typically requires waiting for agreement or confirmation.

Because of these costs, the most reliable distributed systems are not necessarily the ones with the strongest or most complex coordination mechanisms.

Instead, they are often the systems that minimize coordination as much as possible while still preserving correctness.

23. Let the Database Protect Its Own Data

Suppose an application allows users to register with an email address.

A straightforward but naive approach might introduce a distributed lock to prevent duplicate registrations. The flow would look like this:

Acquire Distributed Lock
        |
        v
Check Email Exists
        |
        v
Insert Customer
        |
        v
Release Lock

In this design, the distributed lock ensures that two application instances do not attempt to create the same customer at the same time.

While this approach works in principle, it is often unnecessary in practice. Relational databases already provide strong guarantees about data integrity, including uniqueness constraints. Instead of coordinating at the application level, we can express the business rule directly in the database schema.

A unique constraint makes the intent explicit and enforces it reliably:

CREATE UNIQUE INDEX uk_customer_email
ON customer(email);

With this constraint in place, every application instance can simply attempt the insert operation without any external coordination.

The flow becomes much simpler:

Insert Customer
        |
        |
Unique Constraint
        |
  Success/Reject

There is no distributed lock.
There is no coordination service.
There is no lease management.

The database already owns the data, and therefore it is the most appropriate place to enforce rules about that data.

Whenever correctness depends entirely on the state of a single database, delegating enforcement to the database itself is usually the simplest and most reliable solution.

24. Optimistic Concurrency Instead of Ownership

Some coordination problems arise because multiple application instances attempt to modify the same record at the same time.

At first glance, this seems to require a distributed lock to ensure exclusive access.

However, that assumption is not always correct.

Consider an inventory record:

Product
--------
Quantity = 25
Version = 8

Now imagine two application instances reading and updating this record concurrently.

Both instances start from the same version:

Application A

Version 8
-------------------

Application B

Version 8

Application A performs an update first, and the database successfully applies the change:

Version = 9

When Application B attempts to apply its update, it is still working with the outdated version 8. The database detects this mismatch and rejects the update.

At this point, the application can retry the operation using the latest state from the database.

No distributed lock was required at any point in this process.
No application had to wait for exclusive ownership of the record.

Instead of preventing concurrent work, optimistic concurrency allows concurrent operations to proceed and resolves conflicts only when they actually occur. This approach is particularly effective when conflicts are relatively rare.

It provides high throughput because operations are not blocked unnecessarily. It minimizes coordination overhead between services. It also offers a simple recovery model, where failed updates are retried with fresh state.

This combination of properties is why optimistic concurrency is widely used in modern backend systems.

25. Idempotency Eliminates Many Locking Problems

Consider a scenario where a payment request is received more than once.

This can happen for many reasons in distributed systems.

A client might retry after a timeout, even though the first request already succeeded.

A message broker might deliver the same event more than once.
A network failure might occur after the operation completes but before the response is acknowledged.

One possible solution is to use a distributed lock to ensure the payment is only processed once:

Acquire Lock
      |
Charge Customer
      |
Release Lock

However, there is another approach that avoids coordination entirely by changing the nature of the operation itself.

In this design, every request includes an idempotency key that uniquely identifies the operation:

Payment Request

Idempotency Key
PAY-48291

Before processing the payment, the system checks whether this key has already been used.

The flow becomes:

Receive Request
       |
Already Processed?
       |
   +---+---+
   |       |
 Yes       No
  |         |
Return     Process
Result     Payment

If the request has already been processed, the system simply returns the previous result. If not, it proceeds with the payment and records the key. This makes duplicate requests safe by design.

Instead of coordinating ownership between multiple application instances, the system ensures that repeating the same operation does not change the outcome.

This approach is often more robust than distributed locking because retries are not edge cases in distributed systems—they are expected behavior.

Reliable systems are designed with retries in mind. They do not merely try to prevent them; they make them safe.

26. Partition Ownership Instead of Global Coordination

Consider a payment platform that processes millions of transactions every day.

If every payment required a distributed lock, the system would quickly become a bottleneck due to the overhead of coordination.

A more scalable approach is to partition the work.

Payments A-H
      |
      v
Consumer A
-------------------

Payments I-P
      |
      v
Consumer B
-------------------

Payments Q-Z
      |
      v
Consumer C

In this model, each consumer is responsible for a specific subset of the workload. Each partition has a clear owner, and that ownership is determined by the partitioning strategy rather than dynamic locking.

As a result, applications no longer compete for individual operations. Instead, they operate independently within their assigned partitions. This pattern is commonly used in message brokers and stream processing systems.

Rather than coordinating every single operation globally, the system only coordinates partition ownership. This significantly reduces synchronization overhead and improves scalability.

Partitioning is one of the most effective techniques for reducing the need for distributed coordination in large systems.

27. Sharding Reduces Shared Ownership

A similar idea applies to data storage systems.

Instead of storing all data in a single shared database, the data can be divided into shards.

For example, customer data might be partitioned alphabetically:

Customers A-M
       |
       v
Database A
-------------------

Customers N-Z
       |
       v
Database B

Each shard is responsible for a subset of the data and operates independently.

Most requests only interact with a single shard, which means there is no need for global coordination across the entire dataset. Applications only need to coordinate within the boundaries of a single shard, not across all shards.

By reducing the amount of shared ownership, sharding naturally reduces the need for distributed locking and other global coordination mechanisms.

28. Sometimes Coordination Isn't Necessary

Not all distributed problems require strict exclusivity or ownership. In some cases, concurrent operations can safely proceed without coordination at all.

For example, consider a distributed counter that is updated by multiple application instances.

Instead of preventing concurrent updates, the system can use a data structure designed to merge updates safely. In this model, multiple instances can update the counter at the same time without conflict.

The applications are not competing for control; they are contributing to a shared result. Some distributed data structures are specifically designed to support this kind of behavior.

Their goal is not to prevent concurrent updates, but to ensure that all updates eventually converge to a consistent final state. While the details of these structures are beyond the scope of this discussion, they highlight an important principle in distributed system design.

Coordination should only be introduced when it is truly required for correctness.

29. Choosing the Simplest Correct Solution

As distributed systems grow in complexity, a clear pattern emerges.

Different problems require different coordination strategies, and not all of them require distributed locking.

Using a distributed lock simply because multiple machines are involved often leads to unnecessary complexity and reduced scalability.

A better approach is to choose the simplest mechanism that still guarantees correctness.

Engineering Problem	Recommended Approach	Why It's a Better Choice
Prevent duplicate customer registration	Database Unique Constraint	The database is the source of truth and can enforce uniqueness atomically without external coordination.
Concurrent updates to the same record	Optimistic Locking	Detects write conflicts efficiently while allowing high concurrency and minimal coordination.
Duplicate payment or API requests	Idempotency	Makes retries safe by ensuring repeated requests produce the same outcome.
One instance should execute scheduled jobs	Leader Election	Elects a single coordinator while allowing automatic failover if the leader becomes unavailable.
Exclusive access to a shared resource	Lease + Fencing Tokens	Provides temporary ownership and prevents stale owners from modifying shared state after lease expiration.
Maintain consistent cluster configuration or metadata	Consensus	Ensures all healthy nodes agree on a single, consistent view of shared state despite failures.
Process high-volume event streams or queues	Partition Ownership	Assigns exclusive ownership of partitions to consumers, eliminating the need for global locking.
Distribute ownership of large datasets	Sharding	Reduces shared ownership by partitioning data, minimizing the need for cross-node coordination.
Merge concurrent updates safely without coordination	Conflict-Free Data Structures (CRDTs)	Allows concurrent updates to converge automatically, avoiding locks for specific classes of problems.

One important observation is that distributed locking appears only once in this table.

This is intentional.

Most real-world problems are better solved using mechanisms that are specifically designed for those problems, rather than relying on a general-purpose locking abstraction.

30. Common Misconceptions

There are several misconceptions that frequently appear when discussing distributed locking in system design.

One common belief is that distributed locks guarantee correctness. In reality, this is not always true. Correctness depends not only on acquiring the lock, but also on how the protected resource validates ownership. Without mechanisms like fencing tokens, stale lock holders may still be able to modify shared state incorrectly.

Another misconception is that lease expiration implies the original owner has stopped working. In practice, this is not guaranteed. The application may still be running but temporarily unable to renew the lease due to garbage collection pauses, CPU starvation, or network delays. Distributed systems must reason in terms of timeouts and uncertainty, not absolute state.

It is also often assumed that leader election replaces consensus. This is not the case. Leader election determines which node is responsible for coordination, while consensus determines what the system agrees upon. These are fundamentally different problems that operate at different layers of the system.

Finally, there is a belief that every distributed system requires distributed locking. In reality, many systems do not. Techniques such as unique constraints, optimistic concurrency, idempotent operations, and partition ownership often provide simpler and more scalable solutions.

Final Thoughts

We began this discussion with a simple question: how can multiple application instances ensure that only one of them performs a particular operation?

At first, distributed locking seems like a complete solution. It enforces exclusivity across machines, giving the impression that once a lock is acquired, all other contenders are safely excluded.

However, each approach introduces trade-offs that matter in real systems.

Permanent locks can become stale if the holder crashes, leaving the system blocked indefinitely.

Leases reduce this risk by introducing time-bounded ownership, but they also allow ownership to expire mid-operation, even when work is still valid. Even with leases, stale ownership can still cause incorrect writes unless the system can distinguish between current and outdated owners.

Fencing tokens address this by enforcing monotonic ordering, ensuring only the latest owner can modify shared state and preventing delayed operations from corrupting data.

At a higher level, leader election coordinates long-running responsibilities by ensuring a single active coordinator, while consensus protocols allow multiple nodes to agree on a single consistent state despite failures.

Together, these mechanisms highlight a core truth about distributed systems:

Distributed locking is not really about locking—it is about coordination under partial and unreliable information, where no node has a complete or up-to-date view of the system.

In practice, machines pause, networks delay or drop messages, and failures often look identical to transient slowness. This makes correctness fundamentally harder to reason about.

As a result, correctness is not achieved by removing uncertainty, but by designing systems that remain correct despite it.

The key lesson is simple:

Before introducing distributed locking, first ask whether the problem can be solved without adding coordination at all, since coordination is often the main source of complexity and fragility.

The most resilient distributed systems are not those with the most advanced locking mechanisms, but those that minimize coordination while still preserving correctness under all expected failure modes.

Assisted AI to paraphrase.

Distributed Locking in Practice: Guarantees, Failure Scenarios and Better Alternatives (3/4)

Venkatesan Ramar — Wed, 19 Aug 2026 07:22:08 +0000

15. Distributed Locking Is Only One Coordination Pattern

By this point, we've established an important progression in how distributed systems handle coordination.

A distributed lock is designed to solve a very specific problem: coordinating ownership of a resource across multiple machines. It ensures that only one node can act on a shared resource at a time.

As systems evolved, we introduced leases to handle a different failure mode—abandoned locks caused by crashed or unreachable nodes. Leases ensure that ownership is temporary and automatically expires if not renewed. We then added fencing tokens to address a more subtle issue: stale owners. Even if a node believes it still holds a lock, a higher-numbered token can prevent it from making unsafe updates.

This naturally leads to a broader question.

If distributed locking has become this sophisticated, why do modern distributed systems still rely on concepts such as leader election and consensus?

The answer lies in a key realization: exclusive ownership is not always the problem we're trying to solve.

In many systems, the goal is not to have multiple machines competing for the same resource. Instead, the goal is much simpler: ensure that one machine coordinates the rest.

That difference shifts the entire coordination model.

16. From Ownership to Leadership

Consider a cluster of application instances responsible for scheduling background jobs.

          +-------------+
          | Instance A  |
          +-------------+

          +-------------+
          | Instance B  |
          +-------------+

          +-------------+
          | Instance C  |
          +-------------+

Each instance runs the same code. Every minute, all of them check whether invoices should be generated.

If all instances execute the scheduler independently, duplicate invoices become inevitable.

A straightforward solution is to use a distributed lock. Before running the scheduler, each instance attempts to acquire the lock. The winner proceeds, and the others wait.

This works correctly, but it introduces a repeating pattern of contention.

Minute 1

Acquire Lock
↓
Generate Jobs
↓
Release Lock
-------------------

Minute 2

Acquire Lock
↓
Generate Jobs
↓
Release Lock
-------------------

Minute 3

Acquire Lock
↓
Generate Jobs
↓
Release Lock

Every execution cycle requires coordination. Every minute, the system competes again for the same ownership.

But the real requirement is simpler: only one instance should be responsible for scheduling. That responsibility does not change frequently.

Continuously competing for it introduces unnecessary overhead.

This is where leader election becomes a better fit.

Leadership Is Long-Lived Ownership

Instead of competing for every operation, the cluster elects a single instance to act as the coordinator.

          +-------------+
          | Instance A  |
          +-------------+
                 │
                 │
          Leader Elected
                 │
                 ▼
          +-------------+
          | Leader      |
          | Instance A  |
          +-------------+

          +-------------+
          | Instance B  |
          +-------------+

          +-------------+
          | Instance C  |
          +-------------+

Once elected, the leader takes responsibility for shared tasks such as:

scheduling background jobs
coordinating cluster state
assigning work to nodes
monitoring cluster health
managing shared configuration

The other instances continue serving requests, but they no longer compete for leadership unless a failure occurs.

This changes the coordination model fundamentally.

Instead of repeatedly asking:

"Who owns this operation right now?"
the system asks:
"Who is currently the leader of the cluster?"

Leadership becomes a long-lived role rather than a short-lived lock acquisition.

17. Leaders Can Fail Too

Leader election does not remove failure scenarios. It simply changes how the system responds to them.

Suppose Instance A is currently the leader.

         Leader

      Instance A
           │
           │
      Coordinates Cluster

While performing its responsibilities, Instance A crashes unexpectedly.

At that moment, the cluster loses its coordinator. No other node is actively managing scheduling or coordination.

To recover, the system must elect a new leader.

This process is called leader election.

Before Failure

Leader
Instance A
---------------------
After Failure

Leader
Instance B

Unlike distributed locking, leader election is not triggered for every operation. It only occurs when leadership is lost.

Most of the time, the leader continues operating without interruption, making it a relatively stable coordination role.

Leadership Is Also Temporary

A natural question arises at this point: how does the system know the leader has actually failed?

The honest answer is that it does not know with certainty.

The same uncertainty we saw earlier still applies.

The leader may have crashed.
The network may be partitioned.
The process may be paused.
The machine may be overloaded.

From the perspective of other nodes, all they can observe is that communication has stopped.

Because of this uncertainty, leader election relies on the same foundational mechanisms we have already discussed:

temporary ownership
heartbeats
leases
timeouts

The problem has not disappeared. It has simply shifted from protecting a resource to protecting the leadership role itself.

18. Why Leader Election Is Not Consensus

Engineers often encounter leader election and consensus together, but they solve different problems.

Leader election answers a narrow question:

Which node should coordinate the cluster?

Consensus answers a broader one:

How do multiple nodes agree on shared state despite failures?

Consider a cluster of three nodes maintaining configuration data.

          Node A

          Node B

          Node C

Now imagine a configuration change is introduced. Every node must eventually agree on the same final state.

If one node applies the change while another rejects it, the cluster becomes inconsistent.

Leader election alone cannot solve this. It only determines who proposes or coordinates the change. It does not guarantee agreement.

Consensus is what ensures that all nodes converge on the same decision.

This distinction is subtle but critical.

Leadership assigns responsibility.
Consensus ensures agreement.

19. Coordination Requires Agreement

To make this more concrete, consider a distributed storage system with three nodes maintaining metadata.

The leader decides that a new storage node should join the cluster.

If the update is not applied consistently across all nodes, the system becomes fragmented.

Node A

Storage 1
Storage 2
Storage 3
------------------

Node B

Storage 1
Storage 2
Storage 3
------------------

Node C

Storage 1
Storage 2

Even though a leader exists, the cluster is now inconsistent. One node has a different view of membership than the others.

This demonstrates an important principle: leadership alone is not enough.

Consensus mechanisms exist to ensure that all healthy nodes eventually agree on the same state.

The exact algorithms behind this—such as Paxos or Raft—are complex and deserve their own discussion.

For now, the key architectural insight is:

Leader election determines who coordinates.
Consensus determines what everyone agrees on.

20. Where Modern Coordination Systems Fit

Modern distributed coordination systems often combine all of these concepts into a single platform.

They typically provide:

temporary ownership through leases
fencing via monotonic tokens
leader election
consensus mechanisms
failure detection
cluster membership management

Instead of requiring every application to implement these primitives independently, the coordination system provides them as shared infrastructure.

From an application perspective, the architecture becomes much simpler:

Application
      │
      ▼
Coordination Platform
      │
      ├── Leases
      ├── Leader Election
      ├── Cluster Membership
      ├── Consensus
      └── Coordination Metadata

The application focuses on business logic, while the platform handles correctness under failure conditions.

An important observation here is that the specific technology—whether ZooKeeper, etcd, Consul, or another system—is often less important than understanding the underlying coordination problem.

Choosing a tool before understanding the problem often leads to unnecessary complexity.

21. Choosing the Right Coordination Primitive

Throughout this article, we have explored several coordination mechanisms. While they are often discussed together, each one solves a distinct problem.

Understanding these differences makes system design significantly clearer.

Engineering Problem	Coordination Primitive	Primary Guarantee	Typical Use Cases
Ensure only one node performs a critical operation	Distributed Lock / Lease	Temporary exclusive ownership	Scheduled jobs, file processing, cache refresh, resource ownership
Prevent stale owners from modifying shared state	Fencing Tokens	Rejects operations from older owners	Inventory updates, payment processing, distributed storage, metadata updates
Ensure one active coordinator exists	Leader Election	Single active leader for cluster-wide coordination	Job schedulers, controllers, coordinators, cluster management
Keep multiple nodes in agreement	Consensus	Consistent cluster state despite failures	Cluster membership, configuration management, metadata, distributed coordination
Recover ownership automatically after failures	Leases	Ownership expires unless renewed	Long-running tasks, distributed locks, leadership management
Detect failed coordinators and trigger recovery	Heartbeats + Timeouts	Failure detection based on liveness	Leader monitoring, cluster health, node membership

This table reflects the progression we have followed throughout the article.

We began with distributed locks for shared resource coordination.
We then addressed their limitations with leases.
We fixed stale ownership with fencing tokens.
We improved coordination efficiency with leader election.
Finally, we ensured correctness across nodes with consensus.

Each mechanism builds on the limitations of the previous one. None replaces the others. Instead, they form a layered toolkit for building reliable distributed systems.

The final question in this series naturally follows from here:

Do we actually need distributed locking at all?

In many real-world systems, correctness is achieved through alternative approaches such as optimistic concurrency, idempotent operations, database constraints, partition ownership, or queue-based processing. Understanding when to use these alternatives is often more valuable than implementing a distributed lock itself.

Distributed Locking in Practice: Guarantees, Failure Scenarios and Better Alternatives (2/4)

Venkatesan Ramar — Tue, 18 Aug 2026 09:48:00 +0000

In this article, we'll explore the mechanisms to solve the coordination problem.

8. Introducing Leases

To address the problem of permanent ownership, distributed systems typically replace it with temporary ownership.

This concept is known as a lease.

Instead of granting indefinite control over a resource, the coordination service assigns ownership for a limited period of time.

Rather than stating, “You own this resource until you explicitly release it,” the system instead says, “You own this resource for the next 30 seconds.”

This changes the interaction model significantly.

Acquire Lease
       |
       v
Execute Work
       |
       v
Renew Lease
       |
       v
Continue Processing

As long as the application remains healthy, it periodically renews the lease to maintain ownership. If the application crashes or becomes unresponsive, it can no longer renew the lease. Once the lease duration expires, ownership is automatically revoked.

At that point, another application becomes eligible to acquire the lease and continue the work.

Leases solve a critical problem in distributed systems: they prevent abandoned locks from blocking progress indefinitely. The system can recover automatically without manual intervention.

However, while leases improve availability, they also introduce a new class of subtle and more complex problems.

Leases Depend on Time

To understand the next challenge, assume the lease duration is thirty seconds.

Application A successfully acquires the lease.

Lease Granted

Duration = 30 seconds

After twenty seconds, the JVM begins a long Full Garbage Collection cycle. This pause lasts forty seconds, significantly longer than the lease duration.

The timeline now becomes problematic.

Lease Granted
      |
      |
Processing
      |
      |
GC Pause (40 sec)
      |
      |
Lease Expires

While Application A is paused, the lease expires. During this time, another application requests access to the same resource.

The coordination service observes that the previous lease has expired and therefore grants ownership to Application B.

Application A             Application B

Lease Holder
      |
GC Pause
                           Acquire Lease
                                |
                          Lease Granted

After the garbage collection completes, Application A resumes execution. From its perspective, it simply continues where it left off, still believing it owns the resource.

At the same time, Application B also believes it legitimately owns the resource because it was granted a valid lease by the coordination service.

Neither application is behaving incorrectly. Both are operating based on information that was valid at different points in time.

However, the system now has two active owners for the same resource.

This situation is significantly more dangerous than an abandoned lock because both participants are actively performing work under the assumption of exclusive ownership.

9. Split Brain Without a Network Partition

Many engineers associate split-brain scenarios exclusively with network partitions, where parts of the system become isolated from each other.

However, as the previous example demonstrates, split-brain conditions can occur even without a network failure.

Application A still believes it holds a valid lease, while Application B has legitimately acquired a newer lease from the coordination service.

        Coordination Service

          Lease Expired
                |
      +---------+---------+
      |                   |
      v                   v
Application A      Application B
Believes           Believes
It Owns            It Owns

Both applications continue processing independently.

Importantly, the coordination service is behaving correctly. It expired the old lease and issued a new one based on its rules. Application A is also behaving correctly, because it has not yet observed that its lease has expired. Application B is also correct, because it received a valid lease.

The problem is not incorrect behavior by any single component. The problem is that ownership information has become stale and inconsistent across the system.

This leads to a crucial realization in distributed systems design: acquiring a lease does not guarantee continuous or permanent ownership. Ownership is not a static property; it can change while an application is temporarily unable to observe that change.

Leases solve one important problem by eliminating abandoned locks, but they introduce another, more subtle challenge.

How do we ensure that an application does not continue performing work after it has lost ownership?

Answering this question leads directly to one of the most important concepts in distributed coordination: fencing tokens.

10. Why Leases Alone Cannot Protect Your Data

The previous section ended with an uncomfortable situation.

Application A acquired a lease, but the JVM paused long enough for that lease to expire. During that time, Application B legitimately acquired a new lease. When Application A eventually resumed, both applications believed they owned the same resource.

This leads to a natural question: why doesn't the coordination service simply reject any further requests from Application A once its lease has expired?

The answer is simple, but important. The coordination service has no control over what Application A does after it acquires the lease. Once ownership is granted, the application runs independently and continues performing work on its own.

It cannot intercept every database update, file write, API call, or business operation that the application performs.

To make this concrete, consider a payment processing service:

            Coordination Service
                    |
             Lease Granted
                    |
                    v
             Payment Service
                    |
                    v
             Banking System

After the lease is granted, the payment service communicates directly with the banking system. At this point, the coordination service is no longer in the execution path.

If the lease expires while the payment service is paused, nothing prevents it from continuing to submit payment requests after it resumes. The coordination service may know the lease has expired, but the banking system does not.

This reveals a key insight: a distributed lock controls ownership, but it does not automatically control every operation performed by the owner.

Ownership and Authority Are Different

To understand the problem more clearly, consider a warehouse system that reserves inventory.

Application A acquires a lease and begins processing. While it is still working, the lease expires. At that point, Application B becomes the new owner.

       Lease Owner

     Application A
            ↓
     Lease Expires
            ↓
     Application B

Now imagine Application A resumes execution. Even though it is no longer the owner, it still holds references to in-memory objects, still has an open database connection, and still has network access.

Nothing physically prevents it from continuing to update inventory.

The coordination service cannot reach back in time and undo or block operations that are already in progress.

So while ownership has changed, the authority to modify the shared resource has not been automatically revoked in the running application.

This is why leases alone cannot guarantee correctness in distributed systems.

11. The Missing Piece

Let’s revisit the inventory example in more detail.

The inventory contains a single remaining item. Application A acquires a lease and begins processing. A few seconds later, it pauses unexpectedly. During this pause, the lease expires.

Application B then acquires a new lease and successfully reserves the inventory item.

Several seconds later, Application A resumes and also attempts to reserve the same item.

At this point, the inventory becomes inconsistent.

What is important here is that neither application violated the lease protocol. The coordination service behaved correctly. Both applications followed the rules as designed.

The problem is that the shared resource had no way to distinguish between the current owner and a previous owner that resumed late.

Both requests looked valid.

The missing capability is not another lock or another lease mechanism. The missing capability is a way to determine which owner is newer.

This is exactly the problem that fencing tokens solve.

12. Introducing Fencing Tokens

Instead of only granting a lease, the coordination service also issues a monotonically increasing number along with it.

This number is called a fencing token.

Each time a new lease is granted, the token increases. Every new owner receives a strictly larger value than the previous one.

The sequence might look like this:

Application A
Lease Granted
Token = 101
--------------------
Application B
Lease Granted
Token = 102
--------------------
Application C
Lease Granted
Token = 103

The fencing token represents the freshness of ownership. A larger token always means a more recent owner.

Unlike timestamps, fencing tokens do not depend on clock synchronization. They are simply increasing numbers generated by the coordination system.

This small addition fundamentally changes how safety is enforced in distributed systems.

Why Increasing Numbers Matter

Now consider what happens when Application A pauses after receiving token 101. While it is paused, Application B acquires a lease and receives token 102.

Time

Application A
Acquire Lease
Token 101
      |
      |
GC Pause
      |
      |
Resume
-------------------------

Application B

Acquire Lease
Token 102

Both applications continue running independently.

Without fencing tokens, both would appear equally valid when they attempt to modify the shared resource.

With fencing tokens, every operation now carries proof of ownership.

Application A sends a request like this:

Reserve Inventory

Token = 101

Application B sends:

Reserve Inventory

Token = 102

The resource can now immediately determine which request belongs to the most recent owner.

13. The Resource Must Enforce the Token

This is the point where many explanations stop, and also where many real-world systems fail.

The coordination service generates fencing tokens, but it does not enforce them. The responsibility of validation belongs entirely to the protected resource.

For example, imagine the inventory database keeps track of the highest token it has seen so far.

Initially, the state might be:

Highest Token = 101

When Application B submits its request with token 102, the database compares it with the stored value. Since 102 is greater, the request is accepted, and the state is updated:

Highest Token = 102

Later, Application A resumes and submits a request with token 101. The database performs the same comparison:

101 < 102

Since the incoming token is older than the current highest token, the request is rejected.

This prevents the stale owner from modifying the shared resource.

The key insight is that correctness is enforced by the resource itself, not by the coordination service and not by the lease mechanism.

Why the Coordination Service Cannot Do This

At first glance, it might seem simpler if the coordination service itself rejected stale operations.

However, this is not possible in practice.

The coordination service only participates during lease acquisition. After that point, all business operations happen directly between the application and the resource.

Consider this flow:

Application
      |
Acquire Lease
      |
Coordination Service

Application
      |
      |
      v
Database

Once the lease is granted, every subsequent operation bypasses the coordination service entirely. Because of this, only the database (or the protected resource) can determine whether a request is stale.

This is why fencing tokens must accompany every operation that modifies shared state.

14. Where Fencing Tokens Are Useful

Fencing tokens are useful anywhere multiple distributed processes can modify a shared resource.

Common examples include:

updating inventory systems
processing payment batches
writing to shared storage systems
updating distributed metadata stores
controlling scheduled or background jobs
modifying cluster configuration or leadership state

The underlying technology does not matter. What matters is that multiple actors can attempt to modify the same resource, and correctness depends on ensuring only the newest owner succeeds.

Whenever a system must distinguish between stale and current ownership, fencing tokens provide a reliable solution.

Fencing Tokens Are Not a Replacement for Leases

It is important to understand that leases and fencing tokens solve different problems.

Leases answer the question:

"Who currently owns the resource?"

Fencing tokens answer a different question:

"Is this request coming from the most recent owner?"

A system that uses leases without fencing tokens can still suffer from stale writes. A system that uses fencing tokens without leases has no mechanism for transferring ownership in the first place.

In practice, robust distributed systems use both together.

The lease establishes temporary ownership.

The fencing token allows the resource to verify that ownership before accepting any operation.

This separation of concerns is a key idea behind production systems like ZooKeeper and etcd, which must remain correct even in the presence of failures, pauses, and network partitions.

Distributed Locking in Practice: Guarantees, Failure Scenarios and Better Alternatives (1/4)

Venkatesan Ramar — Mon, 17 Aug 2026 05:41:00 +0000

Distributed systems solve many problems by dividing work across multiple machines. The same characteristic also introduces an entirely new class of problems.

Machines must coordinate.

Unlike threads running inside a single JVM, independent services have no shared memory, no common execution context, and no built-in synchronization mechanism. Every instance makes decisions based on the information available to it at that moment. When several instances attempt to perform the same business operation simultaneously, coordination becomes significantly more difficult than it first appears.

Distributed locking is often introduced as the solution to this problem.

It certainly plays an important role.
It is also one of the most misunderstood concepts in distributed systems.

Many discussions focus on implementing distributed locks using a particular technology. Much less attention is given to understanding what distributed locking actually guarantees, what it cannot guarantee, and why many production systems solve coordination problems without using distributed locks at all.

This article explores distributed locking from an engineering perspective.

Rather than starting with technologies, we'll begin with the coordination problem itself.

As the discussion progresses, each solution will naturally expose its own limitations. Those limitations lead us toward leases, split-brain scenarios, fencing tokens, leader election, consensus, and finally the situations where a distributed lock is not the right solution.

Because distributed locking is ultimately not about acquiring a lock.

It is about keeping distributed systems correct while machines fail, networks become unreliable, and time itself becomes uncertain.

1. The Coordination Problem

Imagine an e-commerce platform during a flash sale. Only one unit of a limited-edition product remains in inventory. Two customers place an order at nearly the same time.

The application is deployed across multiple instances.

                Inventory = 1

          +----------------------+
          |                      |
          v                      v
     Application A          Application B
          |                      |
          |                      |
     Reserve Item           Reserve Item

Both application instances receive the request almost simultaneously.

Both read the inventory.
Both conclude that one unit is available.
Both attempt to reserve it.

The inventory now becomes negative.

No service crashed.
No exception occurred.
The database remained healthy.

Every component behaved exactly as designed.

The failure was not caused by incorrect code. It was caused by the absence of coordination.

Both applications made independent decisions about the same business resource.

This problem appears in many business systems.

Examples include processing the same payment twice, generating duplicate invoices, executing the same scheduled job on multiple servers, refreshing the same cache simultaneously, and assigning the same delivery request to multiple drivers.

Although these scenarios appear unrelated, they all share the same underlying problem.

Multiple independent machines need to agree that only one of them should perform a particular operation. That is coordination problem.

Coordination Is Different From Concurrency

Concurrency is a familiar concept for most Java developers.

Two threads execute simultaneously inside the same JVM. They share memory, they share execution context, and they rely on built-in synchronization primitives provided by the language and runtime. Because of this shared environment, coordination is relatively straightforward: threads can directly block, signal, and enforce mutual exclusion using well-defined mechanisms.

Distributed systems operate under entirely different conditions.

Application instances do not share memory. They cannot directly observe each other's state. Communication happens over a network, which introduces latency, partial failures, and uncertainty. Messages may be delayed, reordered, or lost. Machines may fail independently without affecting others. As a result, every instance has only a partial and potentially outdated view of the overall system.

Imagine three application instances running in different environments.

  +-------------+    +-------------+    +-------------+
  | Application |    | Application |    | Application |
  |      A      |    |      B      |    |      C      |
  +-------------+    +-------------+    +-------------+

Each instance executes independently, making decisions based solely on local information. None of them can infer the state of the others unless explicit communication occurs. This lack of shared state fundamentally changes how synchronization must be designed in distributed systems.

2. Why Traditional Locks Stop Working

Most Java developers are comfortable solving concurrency problems using synchronization primitives such as synchronized blocks or explicit locks.

Consider a simple example of protecting a critical section in a single JVM.

public class InventoryService {

    private final Object lock = new Object();

    public void reserve(String productId) {

        synchronized (lock) {
            // Reserve inventory
        }

    }

}

Inside a single JVM, this approach works exactly as expected. Only one thread can enter the critical section at a time, and the JVM enforces mutual exclusion reliably. The lock effectively serializes access to shared in-memory state.

The problem becomes apparent when the same application is deployed across multiple instances.

Suppose the service runs on two different servers.

          JVM A                     JVM B

     synchronized(lock)      synchronized(lock)

Each JVM maintains its own memory space. Each instance creates its own independent lock object. There is no shared state between them, and neither JVM is aware of the other’s existence. As a result, both threads can enter their respective synchronized blocks simultaneously without any coordination.

The synchronization mechanism has not failed. It has simply been applied outside its intended scope. It only guarantees mutual exclusion within a single process, not across multiple processes.

The same limitation applies to other in-memory primitives such as ReentrantLock, Semaphore, and ReadWriteLock. All of them assume a shared memory model, which distributed systems fundamentally do not provide.

What About Database Locks?

A natural question arises at this point: if multiple application instances share the same database, can the database itself act as a coordination mechanism?

The answer is nuanced.

In some cases, it works well. In others, it is appropriate but limited. In many cases, it is not suitable at all.

Consider a scenario where two applications attempt to update the same customer record. A relational database can enforce row-level locking, ensuring that only one transaction modifies the row at a time.

Application A
       |
Row Locked
       |
Application B Waits

This works effectively because the resource being protected—the database row—already exists within the database’s control domain. The database is naturally responsible for managing consistency of its own data, so using its locking mechanism aligns with its design.

Now consider a different scenario: a scheduled reconciliation job that must run only once per night. The job performs multiple external operations such as downloading files, calling third-party services, generating reports, and uploading results.

None of these operations map cleanly to a single database row. There is no single database entity that represents the job’s execution state in a way that the database can meaningfully lock. The resource requiring coordination exists outside the database’s transactional boundary.

This highlights an important distinction. Database locks are highly effective for protecting database state, but they are not a general-purpose coordination mechanism for distributed workflows that span multiple systems and external dependencies.

Recognizing this boundary prevents overusing database locking as a distributed coordination strategy.

3. What Problem Are We Actually Solving?

At this point, it is easy to assume that the solution is simply a lock that works across machines. However, this intuition only partially captures the real problem.

The actual goal is not the lock itself. The goal is ensuring that only one application instance performs a specific business responsibility at a given time.

This distinction is subtle but critical.

Consider an application responsible for generating monthly customer statements. If three instances of the application start simultaneously, each instance may attempt to perform the same job independently.

Instance A
             |
             |
        Generate Statements

        Instance B
             |
             |
        Generate Statements

        Instance C
             |
             |
        Generate Statements

If all three proceed, the system produces duplicate work and potentially inconsistent results. However, the business requirement is not about locking—it is about ensuring exclusivity of execution.

Only one instance should perform the task.

This requirement is fundamentally about ownership rather than synchronization. The system needs a way to assign temporary responsibility for a task to a single participant.

This pattern appears repeatedly in distributed systems. One instance may need to refresh a shared cache, process a payment batch, act as a leader, execute scheduled tasks, or migrate shared data. In all cases, the requirement is consistent: a single owner must perform a specific responsibility for a defined period of time.

Distributed locking is one possible mechanism to achieve this outcome, but it is not the outcome itself.

4. The First Distributed Lock

To address coordination across multiple machines, a shared coordination service is often introduced. Before performing a critical operation, each application instance requests permission from this service.

The interaction typically follows a simple pattern.

Application
      |
Acquire Lock
      |
      v
Coordination Service
      |
Lock Granted
      |
      v
Execute Work
      |
Release Lock

From a conceptual standpoint, this resembles traditional synchronization. One participant acquires exclusive access, others wait or retry, and only a single instance proceeds with execution.

At first glance, this appears to solve the problem effectively. The coordination service replaces shared memory, and mutual exclusion is enforced across distributed instances. The business requirement of single execution is satisfied.

If distributed systems were perfectly reliable, this model would be sufficient and straightforward.

However, real-world systems introduce complexity that does not exist in single-process environments. Time becomes unreliable. Machines can fail unexpectedly. Network communication can be delayed or interrupted. Processes may pause due to garbage collection or crash entirely.

These factors introduce uncertainty into the system. A lock that was granted moments ago may no longer represent a valid or safe assumption about system state.

As a result, the problem evolves.

It is no longer sufficient to ask, “Who holds the lock?”

The more important question becomes, “Is the current lock holder still alive and capable of safely completing the work?”

This shift in perspective fundamentally changes how distributed locking must be designed and evaluated. It also explains why acquiring a distributed lock is only the beginning of the problem, not its solution.

5. Failure Changes Everything

At the end of the previous section, we arrived at a seemingly simple solution.

Every application instance requests a distributed lock before performing a critical operation.

Only one instance receives the lock, while all other instances are forced to wait until it becomes available again.

From a business perspective, this appears to satisfy the requirement because only one application performs the work at any given time.

If distributed systems behaved like a single JVM, this would be sufficient. In a single process, synchronization primitives provide strong guarantees about mutual exclusion and ownership.

However, distributed systems do not behave like a single JVM.

The difficulty begins the moment we acknowledge a simple reality: machines fail.

More importantly, machines sometimes appear to fail even when they are still running correctly. This distinction fundamentally changes how distributed coordination must be designed.

When a Machine Stops Responding

Let’s assume an application successfully acquires a distributed lock.

Application A
      |
Acquire Lock
      |
Lock Granted
      |
Processing...

At this point, everything appears normal. The system has successfully established exclusive access, and the application begins performing its critical work.

A few seconds later, the application stops responding.

This immediately raises an important question: should another application be allowed to acquire the lock?

At first glance, the answer seems obvious. If the application is not responding, it feels reasonable that another instance should take over and continue the work.

However, distributed systems rarely provide clarity about why a machine stopped responding. There are many possible explanations, and from the outside, they often look identical.

The application may have crashed unexpectedly due to a runtime error or fatal exception. The operating system itself may have restarted the machine as part of maintenance or recovery. The network connection may have been interrupted, isolating the machine even though it is still running. The machine may simply be overloaded, causing extreme delays in processing requests. In some cases, the process may still be alive and functioning correctly, but temporarily unable to communicate with the rest of the system.

From the perspective of other machines, all of these scenarios produce the same observable behavior: the application becomes silent.

This silence is deceptively simple, yet extremely difficult to interpret correctly in a distributed environment.

Failure Is Often Uncertainty

One of the most important differences between concurrent programming and distributed systems is that failures are rarely definitive in distributed environments.

Inside a single JVM, the behavior is clear and deterministic. A thread either holds a lock or it does not. A method either completes successfully or it throws an exception. The system has complete visibility into execution state.

Distributed systems, however, operate with incomplete and delayed information.

Imagine three application instances interacting in a network.

               +-------------+
               | Instance A  |
               +-------------+
                      |
          Communication Lost
                      X
        +-------------+-------------+
        |                           |
        v                           v
 +-------------+             +-------------+
 | Instance B  |             | Instance C  |
 +-------------+             +-------------+

In this scenario, Instances B and C stop receiving messages from Instance A. From their perspective, communication has simply ceased.

There are several possible explanations for this condition. Instance A may have crashed entirely. The network between the instances may have failed or become partitioned. Instance A may still be running but temporarily paused or overloaded. Alternatively, messages may simply be delayed in transit due to congestion or resource constraints.

The key challenge is that the remaining instances cannot immediately distinguish between these possibilities. They only observe the absence of communication, not its cause.

This uncertainty is not an edge case in distributed systems; it is the default operating condition. As a result, engineering decisions must often be made without complete or reliable information.

6. Time Introduces New Problems

Now consider a different scenario. Application A successfully acquires a distributed lock and begins processing a large batch of financial transactions.

Initially, everything proceeds as expected. The system is stable, and the application is actively performing work.

However, after a short period, the JVM triggers a Full Garbage Collection cycle. During this process, the application is paused while memory is being reclaimed and reorganized.

Acquire Lock
      |
      v
Begin Processing
      |
      v
Full GC Pause
      |
      |
      |
      v
Resume Processing

From the perspective of the application itself, nothing unusual has occurred. Execution has simply been paused temporarily and will resume once garbage collection completes.

From the perspective of the rest of the system, however, the situation looks very different. During the pause, the application is unable to respond to requests, send heartbeats, or communicate its state. For several seconds, it appears completely unresponsive.

Importantly, the application is not dead. It is still running. It is simply not making progress.

Distributed systems struggle with this distinction because external observers cannot easily determine whether a system is paused or permanently failed.

A Pause Can Look Like a Failure

This ambiguity often surprises engineers who are new to distributed systems.

A useful analogy is waiting for a colleague to respond to an important message. If several minutes pass without a reply, multiple interpretations become possible. The colleague may be busy with other work. Their phone may be turned off or disconnected. They may have lost internet connectivity. In the worst case, they may no longer be available at all.

Without additional context, all of these explanations remain equally plausible.

Distributed systems face the same problem at scale. A temporary pause caused by garbage collection, heavy CPU contention, thread starvation, or resource exhaustion can appear identical to a complete system failure.

Because of this ambiguity, distributed systems cannot rely on perfect failure detection. There is no reliable mechanism that can always distinguish between a slow system and a dead one.

Instead, systems must make decisions based on incomplete and sometimes misleading information.

7. Locks Cannot Last Forever

To understand why distributed locking is fundamentally different from local locking, consider what would happen if a coordination service granted locks indefinitely.

In that model, the interaction would look familiar:

Acquire Lock
      |
      v
     Work
      |
      v
Release Lock

This mirrors traditional synchronization inside a single JVM, where a thread holds a lock until it explicitly releases it.

Now revisit a failure scenario. Application A acquires the lock and begins processing. Shortly afterward, the machine hosting the application crashes unexpectedly.

Application A
      |
Acquire Lock
      |
Machine Crash

Because the application crashes, it never reaches the step where it releases the lock.

Release Lock

As a result, the lock remains permanently held. No other application can acquire it, and every other instance is forced to wait indefinitely.

The system effectively stops making progress.

This illustrates a critical problem: in distributed systems, permanent ownership is dangerous because failures are not rare exceptions—they are expected events.

To maintain availability, the system must be able to recover from abandoned locks automatically.

Assisted AI to paraphrase.

Building Reliable Event-Driven Systems: Event Schemas, Versioning, Contract Testing and Events vs Commands (5/5)

Venkatesan Ramar — Wed, 22 Jul 2026 08:50:00 +0000

In this final article, we're going to explore Production practices

22. Events Are Delivered, Not Guaranteed

A common misconception in event-driven architecture is that publishing an event guarantees it will be processed exactly once.

Production systems rarely provide that guarantee.

Events may be:

delivered multiple times
delayed
delivered out of order
temporarily unavailable
replayed long after publication

Reliable systems are not built on ideal conditions. They assume these scenarios will occur.

As a result, designing resilient consumers is just as important as designing reliable producers.

Duplicate Delivery Is Normal

Consider the following workflow.

OrderConfirmed
       |
       v
Inventory Service
       |
Inventory Reserved

The Inventory Service processes the event successfully. Before acknowledging completion, it crashes.

After recovery, the event is delivered again.

From the messaging system’s perspective, this is correct behavior. From the application’s perspective, the same business event has arrived twice.

Without safeguards, inventory may be reserved twice.

Duplicate delivery is not an edge case. It is expected behavior in distributed systems.

23. Idempotency Makes Event Processing Safe

Idempotency ensures that executing the same operation multiple times produces the same result.

For event consumers, this means processing the same event repeatedly should not create duplicate business outcomes.

Using Event Identifiers

Each event should include an eventId.

Consumers track processed events:

Processed Events
----------------
E-1001
E-1002
E-1003

When a new event arrives, say: eventId = E-1002

The consumer checks if it has already processed it.

If yes, ignore it
If no, process it

Flow:

Receive Event
      |
      v
Already Processed?
      |
 +----+----+
 |         |
Yes        No
 |          |
Ignore   Process Event
            |
            v
      Record eventId

This approach ensures correctness even with duplicate delivery.

Idempotency Belongs to Business Logic

Infrastructure can reduce duplicates but cannot eliminate them.

Idempotency must be enforced at the application level.

Example: a payment service must never charge a customer twice for the same event.

This guarantee belongs to business logic, not messaging infrastructure.

24. Ordering Cannot Always Be Assumed

Developers often assume events arrive in the order they were published.

In reality, delivery order can change due to:

network latency
retries
parallel processing
independent consumers

Expected sequence:

OrderCreated
      |
OrderConfirmed
      |
OrderShipped

A consumer might observe:

OrderConfirmed
      |
OrderCreated
      |
OrderShipped

The producer behaved correctly. The delivery order changed.

Consumers should not rely on strict ordering unless explicitly guaranteed.

Design Consumers to Tolerate Reordering

Consumers should validate state before processing.

Example:

Receive `OrderShipped`
         |
   Order Exists?

If required state is missing:

retry later
delay processing
move the event for later handling

This approach handles temporary inconsistencies without assuming perfect ordering.

25. Correlation Makes Distributed Systems Understandable

A single user action can trigger multiple events across services.

Consider a sample flow:

Create Order
      |
OrderConfirmed
      |
InventoryReserved
      |
PaymentCompleted
      |
ShipmentCreated
      |
InvoiceGenerated

Without correlation, these events appear unrelated. But, with a shared identifier like REQ-98451 the entire workflow becomes traceable.

Correlation improves observability without affecting business logic.

26. Events Become Part of Your Audit Trail

Events naturally capture business history.

Example lifecycle:

OrderCreated
      |
OrderConfirmed
      |
PaymentCompleted
      |
ShipmentCreated
      |
OrderDelivered

This sequence provides a complete record of what happened. Because events represent completed facts, they should never be modified.

History must remain immutable.

27. Observability Should Include Events

Traditional systems usually track:

request latency
response time
database queries

Event-driven systems require additional metrics:

published events
consumed events
failed processing
retry count
duplicate events
processing latency
dead-letter events

Without these, failures may go unnoticed until business issues appear.

Observability must include event flows, not just services.

Tracing Event Flows

Distributed tracing becomes more effective with correlation identifiers.

Example:

Customer Request
        |
Order Service
        |
Inventory Service
        |
Payment Service
        |
Shipping Service

Tracing allows engineers to follow a complete business workflow instead of isolated logs.

This significantly improves debugging and incident analysis.

28. A Practical Checklist

Reliable event-driven systems consistently follow these practices.

Design
✓ Events represent business facts
✓ Events expose business concepts, not internal models
✓ Payloads contain meaningful business data
✓ Technical metadata is separated from business data

Evolution
✓ Schemas evolve carefully
✓ Breaking changes are avoided
✓ Optional fields are preferred over removal
✓ Compatibility is continuously validated

Reliability
✓ Consumers are idempotent
✓ Duplicate delivery is expected
✓ Ordering is not assumed
✓ Correlation identifiers are included

Operations
✓ Contracts are validated automatically
✓ Event processing is observable
✓ Historical events remain immutable
✓ Producers and consumers evolve independently

Final Thoughts

Event-driven architecture is often described as a messaging pattern. In practice, it is a contract-driven architecture.

Messaging transports events. Contracts enable independent evolution.

Reliable systems are not defined by tools or frameworks. They are defined by disciplined practices:

designing events as business contracts
evolving schemas safely
validating contracts continuously
building resilient consumers

These principles apply regardless of technology.

When applied consistently, events become more than messages. They become stable, long-lived contracts that support scalable and maintainable distributed systems.

Assisted AI to paraphrase.

Building Reliable Event-Driven Systems: Event Schemas, Versioning, Contract Testing and Events vs Commands (4/5)

Venkatesan Ramar — Mon, 20 Jul 2026 09:00:00 +0000

In this article, we're going to explore Contract Testing

Contract testing ensures that event contracts remain stable as systems evolve. Producers receive immediate feedback when changes affect consumers, and consumers gain confidence in the data they process.

16. Why Contract Testing Matters

Events act as public contracts, and those contracts evolve as systems grow. The key question is how a producer can verify that a schema change does not break its consumers.

Many teams rely on integration testing for this purpose. While useful, integration tests only confirm that systems can communicate. They do not guarantee that all consumers still understand the event contract.

Consider an Order Service publishing an OrderConfirmed event consumed by multiple services:

                  OrderConfirmed
                         |
      +------------------+-------------------+
      |                  |                   |
      v                  v                   v
 Inventory          Billing          Notification

If the producer renames a field:
{ "customerId": "CUS-501" }
to:
{ "accountId": "CUS-501" }

everything may still appear correct. The code compiles, tests pass, and deployment succeeds. However, the Billing Service may fail at runtime because it still expects customerId. The producer had no visibility into this dependency.

The issue is not deployment failure but contract violation. Contract testing exists to detect such issues before they reach production.

Integration Tests Cannot Protect Unknown Consumers

Integration tests typically validate interactions between known systems:

Order Service
      |
      v
Inventory Service

In this setup, both systems are controlled and predictable. Event-driven systems differ because consumers are loosely coupled and may not be known to the producer.

A new Analytics Service might start consuming OrderConfirmed months later. The producer cannot manually validate every consumer. Contract testing addresses this by validating expectations rather than communication.

17. Consumer-Driven Contracts

Traditional API testing starts with the producer defining the contract. Consumers adapt to it. Consumer-driven contract testing reverses this approach.

Consumers define their expectations, and producers verify that they continue to meet them. This model works well in event-driven systems where consumers evolve independently.

Thinking From the Consumer's Perspective

Different consumers often require different subsets of data. For example, the Billing Service may need:

{
    "orderId": "ORD-1001",
    "customerId": "CUS-501",
    "totalAmount": 249.99
}

The Notification Service may require:

{
    "orderId": "ORD-1001",
    "customerId": "CUS-501",
    "customerEmail": "alice@example.com"
}

The producer does not need to understand each consumer’s logic. It only needs to satisfy their declared contracts. This encourages careful evolution and explicit documentation of expectations.

A Contract Is More Than Field Names

Contracts define more than structure. They also capture meaning and constraints.

For example:
{ "orderId": "ORD-1001", "totalAmount": 249.99 }

A robust contract may specify:
orderId must exist and not be empty
totalAmount must be numeric
totalAmount must be greater than zero

These rules ensure data integrity and provide stronger guarantees for both producers and consumers.

18. Contract Testing in Practice

Consider an Order Service publishing:

{
    "eventType": "OrderConfirmed",
    "eventVersion": "1.0",
    "payload": {
        "orderId": "ORD-1001",
        "customerId": "CUS-501",
        "currency": "USD",
        "totalAmount": 249.99
    }
}

Consumers depend on different fields:
Billing: orderId, customerId, totalAmount
Inventory: orderId
Notification: customerId

If the producer changes the schema:

{
    "payload": {
        "orderId": "ORD-1001",
        "accountId": "CUS-501",
        "currency": "USD",
        "totalAmount": 249.99
    }
}

the system may still compile and deploy. However, contract validation fails because consumer expectations are no longer met. The issue is detected before release, preventing runtime failures.

The Development Workflow

Contract testing integrates into the delivery pipeline:

Consumer Defines Contract
          |
          v
Producer Validates Contract
          |
          v
   Build Succeeds
          |
          v
       Deploy

Every schema change is validated during the build process, ensuring compatibility before deployment.

19. Using JSON Schema to Describe Events

JSON Schema provides a structured way to define event contracts. It acts as an executable specification shared by producers and consumers.

Consider a sample schema:

{
  "type": "object",
  "required": [
    "orderId",
    "customerId",
    "totalAmount"
  ],
  "properties": {
    "orderId": {
      "type": "string"
    },
    "customerId": {
      "type": "string"
    },
    "totalAmount": {
      "type": "number"
    }
  }
}

This schema defines required fields, types, and structure. Additional constraints can include string lengths, numeric ranges, formats, and enumerations. Unlike static documentation, schemas can be validated automatically.

Validation During Publishing

Producers can validate events before publishing:

OrderConfirmedEvent event = ...

validator.validate(event);

eventPublisher.publish(event);

Invalid events are rejected early, preventing faulty data from entering the system.

Validation During Consumption

Consumers can also validate incoming events:

OrderConfirmedEvent event = ...

validator.validate(event);

process(event);

This ensures that only valid data reaches business logic, improving reliability.

20. Schema Registries and Centralized Contracts

As systems scale, managing schemas across multiple repositories becomes difficult. Teams may define similar events differently, leading to inconsistencies.

A centralized schema repository addresses this:

              Event Schemas
                    |
       +------------+------------+
       |                         |
       v                         v
 Producers                  Consumers

This repository becomes the single source of truth. Producers publish against registered schemas, and consumers validate against them.

Why Centralized Schemas Help

Centralized schema management improves consistency and visibility. Contracts become discoverable, version history is preserved, and compatibility rules can be enforced automatically.

Teams spend less time interpreting payloads and more time building features. The contract becomes a shared organizational asset rather than an isolated implementation detail.

21. Common Contract Testing Mistakes

Several common issues arise when adopting contract testing.

Treating Documentation as the Contract

Documentation often becomes outdated as implementations change. Consumers may rely on incorrect assumptions. Executable contracts remain synchronized with the system and provide reliable validation.

Testing Only Happy Paths

Contracts should cover more than valid scenarios. They should include:

missing required fields
invalid data types
unexpected values
optional fields
deprecated fields

This ensures consumers handle both valid and invalid inputs correctly.

Ignoring Backward Compatibility

Passing current contracts does not guarantee future compatibility. Schema evolution must be validated alongside contract correctness to avoid breaking existing consumers.

Assuming Producers Own the Contract

Although producers publish events, contracts are shared responsibilities. Both producers and consumers must maintain and validate them.

Reliable event-driven systems depend on well-defined, continuously validated contracts.

In the next part, we will explore production practices like idempotency, event ordering, duplicate handling, correlation and observability.

Building Reliable Event-Driven Systems: Event Schemas, Versioning, Contract Testing and Events vs Commands (3/5)

Venkatesan Ramar — Fri, 17 Jul 2026 09:27:53 +0000

In this article, we're going to explore Event Schema evolution with Event versioning

10. Event Schemas Will Eventually Change

No event schema stays the same forever. As businesses grow, regulations shift, products gain new features, and processes become more complex, the data shared between services must evolve as well. This evolution is not optional—it is a natural consequence of a system adapting to changing requirements.

Many teams initially assume they can simply update an event whenever needed. This assumption may hold when there is only one producer and one consumer, but real-world systems rarely remain that simple. Over time, multiple consumers emerge, each with its own responsibilities and release cycles.

A typical system often looks like this:

                  OrderConfirmed
                         |
      +------------------+-------------------+
      |                  |                   |
      v                  v                   v
 Inventory          Billing          Notification
      |
      v
 Analytics
      |
      v
 Customer Insights

Each consumer evolves independently. Some services may deploy updates weekly, while others might release changes quarterly. In some cases, consumers may even belong to external teams with entirely different priorities and timelines. Because of this, producers cannot assume that all consumers will upgrade simultaneously.

Schema evolution, therefore, is not just about modifying data structures. It is fundamentally about maintaining compatibility across independently evolving systems.

Compatibility Is More Important Than Version Numbers

When discussing schema evolution, teams often focus immediately on versioning. While versioning is useful, compatibility is far more critical. Without compatibility, versioning alone cannot prevent system breakage.

Consider the following event:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501",
  "totalAmount": 249.99
}

Now imagine a new requirement introduces currency. One approach might replace the existing field entirely:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501",
  "amount": {
      "value": 249.99,
      "currency": "USD"
  }
}

Although the data model has improved, this change breaks every existing consumer that depends on the original structure. The producer has evolved, but the contract has not been preserved.

The goal of schema evolution is to allow both producers and consumers to evolve independently without causing disruptions.

11. Backward and Forward Compatibility

Compatibility is often described using formal definitions, but a practical understanding is more useful when designing real systems.

Backward Compatibility

Backward compatibility ensures that existing consumers continue to function when producers introduce newer versions of events. This is one of the most important principles in event-driven systems.

Consider Version 1 of an event:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501"
}

Now Version 2 introduces an additional field:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501",
  "currency": "USD"
}

Older consumers can safely ignore the new field, allowing them to continue operating without modification. This makes adding optional fields one of the safest ways to evolve a schema. In contrast, removing fields is far more risky because it can break existing consumers.

Forward Compatibility

Forward compatibility addresses the opposite scenario, where newer consumers must handle older events produced by systems that have not yet been upgraded.

For example, a new consumer might expect:

{
  "orderId": "...",
  "customerId": "...",
  "currency": "USD"
}

However, older producers may still emit:

{
  "orderId": "...",
  "customerId": "..."
}

In this case, consumers must be designed to handle missing fields gracefully. They might use default values, leave fields empty, or apply fallback logic. This approach ensures that consumers remain resilient even when the system evolves asynchronously.

Consumers should never assume that every field will always be present, as distributed systems rarely evolve in perfect synchronization.

Compatibility Is a Team Discipline

Most compatibility issues arise not from technical limitations but from incorrect assumptions. A common example is the belief that all consumers have already upgraded to the latest version.

In practice, this assumption is rarely valid. Independent deployment is one of the key advantages of microservices, and compatibility is what preserves that independence. Without it, teams become tightly coupled, and deployments require coordination, defeating the purpose of a distributed architecture.

12. Safe Schema Evolution

Schema evolution is ultimately about maintaining trust between producers and consumers. Producers must continue evolving to meet business needs, while consumers must retain the freedom to upgrade on their own timelines.

Not all schema changes carry the same level of risk. Some changes are generally safe and can be introduced with minimal impact, while others are inherently breaking and require careful planning.

Understanding the difference between these types of changes is essential for preventing production failures.

Adding Optional Fields

Adding optional fields is usually a safe way to evolve a schema. It allows new functionality to be introduced without disrupting existing consumers.

For example:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501"
}

can evolve into:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501",
  "currency": "USD"
}

Older consumers will ignore the new field, while newer consumers can take advantage of it. This approach supports gradual adoption and minimizes risk.

Removing Fields

Removing fields is significantly more dangerous. If any consumer depends on a field, its removal will cause failures.

For instance, if a Billing service relies on:
{ "totalAmount": 249.99 }
and that field is removed, the service will break. Because it is often difficult to know all consumers of an event, it is safest to assume that every published field is in use somewhere.

Renaming Fields

Renaming fields may appear harmless, but it effectively behaves like removing one field and adding another. This makes it a breaking change.

For example:
{ "customerId": "CUS-501" }
changing to:
{ "accountId": "CUS-501" }
will cause existing consumers to fail because they no longer recognize the expected field. Renaming should therefore be treated with the same caution as removing fields.

Changing Data Types

Changing the data type of a field can also introduce subtle but serious issues. Even if serialization succeeds, consumers may fail when processing the data.

For example:
{ "quantity": 5 }
becoming:
{ "quantity": "5" }
may not immediately cause errors during transmission, but it can break downstream logic that expects a numeric value. Type changes should be treated as breaking changes and handled carefully.

13. Versioning Strategies

There are situations where compatibility alone is not sufficient, and the contract must change in a way that cannot be made backward-compatible. In such cases, versioning becomes necessary.

Version Inside the Event

One approach is to include version information directly within the event metadata:

{
   "eventType": "OrderConfirmed",
   "eventVersion": "2.0",
   "payload": {
      ...
   }
}

This allows consumers to adjust their behavior based on the version while keeping the event name consistent. It provides flexibility but requires consumers to handle multiple versions within their logic.

Different Event Types

Another approach is to define separate event types for each version:
OrderConfirmedV1 OrderConfirmedV2

In this model, consumers subscribe only to the versions they support. The producer may need to maintain multiple versions during the transition period, but this approach simplifies consumer logic by avoiding conditional handling within a single event type.

Which Strategy Is Better?

There is no universally correct choice between these strategies. Embedding versions keeps naming simpler, while separate event types make changes more explicit.

The most important factor is consistency. Teams should adopt a standard approach across services to reduce confusion and operational complexity.

14. Avoid Breaking Changes Whenever Possible

Breaking changes introduce tight coupling between services by forcing coordinated deployments. Instead of allowing independent evolution, they create dependencies that can slow down development and increase risk.

Consider the following structure:

Producer
    |
    +-----------------------------+
    |      |      |      |        |
    v      v      v      v        v
Service Service Service Service Service
   A       B       C       D       E

If the producer removes a field, every consumer must update before the change can be safely deployed. Deployment order becomes critical, and the independence of services is lost.

Even a single breaking change can undermine the benefits of an event-driven architecture.

Deprecation Is Usually Better Than Removal

When a field becomes obsolete, it is better to deprecate it rather than remove it immediately.
{ "legacyField": "..." }

Keeping the field while marking it as deprecated allows consumers time to migrate at their own pace. Once all consumers have transitioned away from the field, it can be safely removed.

This gradual approach reduces disruption and maintains system stability.

15. Common Versioning Mistakes

Certain mistakes appear frequently when teams manage schema evolution.

Treating Events Like Internal DTOs

Internal data transfer objects (DTOs) often change rapidly as implementation details evolve. Public event contracts, however, should be treated with much greater care.

They represent agreements between services and should not be modified casually.

Releasing Breaking Changes Without Visibility

Producers often lack visibility into who consumes their events. Making breaking changes without understanding downstream dependencies introduces significant risk.

Contract testing can help address this issue by providing insight into how events are used.

Versioning Every Small Change

Some teams create new versions for every minor change:

OrderConfirmedV2 OrderConfirmedV3 OrderConfirmedV4

In many cases, this is unnecessary. Adding optional fields often preserves compatibility without requiring a new version. Excessive versioning can make systems harder to maintain and understand.

Forgetting That Old Events Continue to Exist

Events are often stored for long periods and may be replayed for analytics, auditing, or recovery purposes. Schema evolution must account for both new and historical events.

Even if the schema changes, historical data remains unchanged. Systems must be able to handle both.

In the next section, we will explore contract testing and how it helps validate these assumptions before changes reach production.

Building Reliable Event-Driven Systems: Event Schemas, Versioning, Contract Testing and Events vs Commands (2/5)

Venkatesan Ramar — Wed, 15 Jul 2026 05:41:00 +0000

In this article, we're going to explore Event Schemas.

5. Designing Event Schemas That Age Well

Once a team adopts event-driven architecture, the event schema quickly becomes one of the most critical design artifacts in the system. Unlike an internal Java class that is used within a single codebase, an event schema is consumed by multiple independent services. These services may be developed, deployed, and maintained by different teams, often evolving at different speeds. As a result, every field included in an event effectively becomes part of a contract that consumers may rely on.

Changing that contract later is rarely as simple as modifying a Java object. While internal models can evolve freely, event schemas must remain stable over time. Good schemas are designed to evolve gracefully, allowing systems to grow without breaking existing consumers. Poorly designed schemas, on the other hand, tend to accumulate compatibility issues that become increasingly difficult to manage.

At the core of this challenge is a simple but powerful question:

Is the event describing a business fact or exposing the producer's implementation?

Events Should Represent Business Concepts

Consider an Order Service that has just confirmed an order. One way to represent this event is by focusing on the business outcome:

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501",
  "status": "CONFIRMED",
  "totalAmount": 249.99
}

Another approach might expose internal structures:

{
  "id": "ORD-1001",
  "customerEntity": {
      ...
  },
  "orderAggregate": {
      ...
  },
  "hibernateVersion": 12
}

Although both events may contain similar information, only the first represents a stable business contract. The second leaks internal implementation details that consumers should not depend on. Over time, such exposure creates tight coupling and makes evolution difficult.

A useful guideline is:

Design events for consumers, not for producers.

The producer already understands its internal model. Consumers only need clear, meaningful business information.

Don't Serialize Your Domain Model

A common mistake is publishing JPA entities directly as events. For example:

eventPublisher.publish(orderEntity);

This approach may seem convenient because it avoids creating additional classes. However, it tightly couples consumers to the producer’s internal structure. Any change in the domain model can unintentionally break consumers.

Consider an initial model:

public class Order {
    private Customer customer;
}

Later, the model evolves:

public class Order {
    private CustomerAccount customerAccount;
}

From a business perspective, nothing has changed. However, the event payload has changed, potentially breaking consumers. This happens because the event contract was tied directly to the internal model.

A better approach is to define dedicated event models:

public record OrderConfirmedEvent(
    String orderId,
    String customerId,
    BigDecimal totalAmount
) {}

This separation ensures that the event contract remains stable even as the internal model evolves. Both can change independently without affecting each other.

6. Designing Event Payloads

Every event answers a specific business question. The schema should provide enough information for consumers to understand that answer clearly, without exposing unnecessary implementation details. Striking this balance is one of the most important design decisions in event-driven systems.

Include Business Information

Consider a minimal event:

{
    "orderId": "ORD-1001"
}

While technically valid, it is not very useful. Consumers will likely need additional information, forcing them to make extra API calls:

OrderConfirmed Event
         |
         v
Inventory Service
         |
         v
GET /orders/ORD-1001

If multiple consumers follow this pattern, a single event can trigger multiple network requests. This increases system load and introduces unnecessary coupling between services.

Avoid Including Everything

At the other extreme, some events include too much information:

{
    "order": {
        ...
    },
    "customer": {
        ...
    },
    "payment": {
        ...
    },
    "inventory": {
        ...
    },
    "shipment": {
        ...
    }
}

Large payloads can lead to higher network usage, increased serialization costs, and tighter coupling between services. They also make schema evolution more difficult, as changes in one part of the payload may affect multiple consumers.

Design Around Business Needs

A practical approach is to ask:

What information should every consumer reasonably expect to receive?

For an OrderConfirmed event, a balanced payload might look like this:

{
    "orderId": "ORD-1001",
    "customerId": "CUS-501",
    "orderDate": "2026-07-01T10:15:30Z",
    "currency": "USD",
    "totalAmount": 249.99
}

This provides essential business context without overwhelming consumers. Those who need additional details can fetch them independently, while others remain unaffected.

7. Event Metadata Matters

While developers often focus on the payload, metadata plays an equally important role in production systems. Metadata provides critical context that helps systems understand how to process and trace events.

It enables systems to determine when an event occurred, where it originated, how it should be tracked, and whether it has already been processed. Without this information, operating and debugging event-driven systems becomes significantly more challenging.

Business Data vs Technical Metadata

Business data should reside in the payload, while technical details belong in metadata. For example:

{
  "eventId": "8c1e6d12",
  "eventType": "OrderConfirmed",
  "eventVersion": "1.0",
  "occurredAt": "2026-07-01T10:15:30Z",
  "correlationId": "REQ-98451",
  "payload": {
      "orderId": "ORD-1001",
      "customerId": "CUS-501",
      "totalAmount": 249.99
  }
}

This separation improves clarity and makes it easier to evolve both business data and technical metadata independently.

Event Identifier

Every event should include a unique identifier: eventId

This identifier is essential for de-duplication, ensuring idempotent processing, enabling tracing, and supporting auditing. It also simplifies handling scenarios where events may be delivered multiple times.

Correlation Identifier

In distributed systems, workflows often span multiple services:

Create Order
      |
Reserve Inventory
      |
Process Payment
      |
Create Shipment

Each step may produce additional events. A correlation identifier links these events together:

Correlation ID REQ-98451

This makes it much easier to trace workflows and debug issues in production environments.

Event Timestamp

Events should record when the business action occurred, not when the event was received. These timestamps can differ due to network delays, retries, or temporary failures. Keeping business time separate from delivery time ensures accurate interpretation of events.

8. Naming Events Consistently

Naming may seem like a minor detail, but it becomes increasingly important as systems grow. Large organizations may produce hundreds of event types, and consistency helps maintain clarity and usability across teams.

Prefer Past-Tense Business Events

Effective event names describe completed business actions:

CustomerRegistered OrderConfirmed InventoryReserved PaymentCompleted ShipmentCreated

These names clearly communicate what has happened, making them easy for consumers to understand.

Avoid CRUD-Oriented Events

Generic names such as:

OrderUpdated CustomerModified ProductChanged

lack clarity. They do not explain what changed or why, forcing consumers to inspect the payload for meaning. Event names should convey intent directly.

Keep Names Business-Oriented

Avoid technical or implementation-focused names:

DatabaseUpdated RowInserted EntitySaved JpaEntityUpdated

These describe internal processes rather than business outcomes. Consumers care about what happened in the business domain, not how it was implemented.

9. Common Schema Design Mistakes

Despite differences in technology, many event-driven systems encounter similar design issues. Recognizing these common mistakes can help teams avoid long-term problems.

Publishing Internal Objects

Internal models change frequently, while public contracts should remain stable. Mixing the two leads to fragile systems. Keeping them separate ensures better maintainability.

Making Events Too Generic

An event like: OrderUpdated can represent many different actions, making it difficult for consumers to interpret. More specific events provide clearer intent:
OrderConfirmed OrderCancelled OrderRefunded

This clarity simplifies consumer logic and improves overall system understanding.

Missing Metadata

Without essential metadata such as event identifiers, timestamps, and correlation IDs, troubleshooting becomes significantly harder. Operational concerns should be considered from the beginning of event design.

Designing for Today's Consumers

Many producers design events based only on current consumers, overlooking future needs. A better approach is:

Design events as though the next consumer has not been written yet.

This mindset encourages more flexible and future-proof designs.

Practical Rule of Thumb

Well-designed schemas remain clear and understandable long after they are introduced. They focus on business facts rather than implementation details, separate metadata from payload data, and provide sufficient context without unnecessary complexity.

In the next part, we will explore schema evolution, event versioning, backward compatibility, and strategies that allow producers and consumers to evolve independently without disrupting production systems.

Assisted AI to paraphrase the content.

Building Reliable Event-Driven Systems: Event Schemas, Versioning, Contract Testing and Events vs Commands (1/5)

Venkatesan Ramar — Tue, 14 Jul 2026 05:00:00 +0000

As part-1 of a multi-part series, in this article we'll explore why and where event-driven systems fail and foundational concepts.

Distributed systems have become considerably easier to build than they were a decade ago. Modern frameworks allow us to publish events with only a few lines of code, cloud platforms provide fully managed messaging services, and frameworks like Spring Boot makes asynchronous communication feel almost effortless. Because of this, many teams successfully adopt event-driven architectures. Unfortunately, publishing events is usually the easiest part of the journey.

Designing events that remain reliable for years is considerably more difficult. Most production problems in event-driven systems are rarely caused by the messaging infrastructure itself. Instead, they originate from architectural questions such as:

What information an event should contain?
Whether an event schema can evolve without breaking existing consumers?
How new services safely consume old events?
When a service should publish an event instead of sending a command?
How producers can know they haven't broken downstream consumers?

These questions determine whether an event-driven system remains maintainable as more services, teams, and business capabilities are added.

This article explores the practices that make event-driven systems resilient over time. We focus on the contracts that services exchange with each other. In event-driven architecture, events become public APIs, and unlike REST APIs, those APIs are usually consumed by systems the producer does not directly control. It makes compatibility one of the most important design concerns.

1. Event-Driven Architecture Is Really About Contracts

Many developers describe event-driven architecture as services communicating through events. That description is correct, but it is also incomplete. Events are more than messages moving between services—every published event represents a contract.

Suppose an Order Service publishes the following event.

{
  "orderId": "ORD-1001",
  "customerId": "CUS-501",
  "status": "CONFIRMED",
  "totalAmount": 249.99
}

Several services subscribe to it.

                 OrderConfirmed
                        |
        +---------------+---------------+
        |               |               |
        v               v               v
 Inventory Service  Billing Service  Notification Service

The producer may know about three consumers today. Six months later, another team builds additional services such as Analytics, Recommendation, and Customer Loyalty. The producer does not need to change; the consumers simply subscribe. This loose coupling is one of the biggest strengths of event-driven systems, but it is also one of their biggest challenges.

The producer no longer knows who depends on the event, which makes changing the event significantly more complicated than changing an internal Java object.

Events Are Public APIs

Most teams treat REST APIs very carefully. Before removing a field, they consider existing clients, API versions, backward compatibility, and migration plans. Events deserve exactly the same level of discipline.

Once an event is published, it becomes part of the public interface of the service. Removing a field from an event can break downstream systems just as easily as removing a field from a REST response. The difference is that consumers are often invisible. A REST API usually has documented clients, while an event may have consumers owned by completely different teams, some of which may not even exist when the producer is originally developed.

Thinking of events as contracts fundamentally changes how they should be designed.

2. Why Event Design Matters More Than Event Publishing

Publishing an event is a technical task, but designing an event is an architectural task. Many event-driven projects begin with something like this:

public class Order {

    private Long id;
    private Customer customer;
    private List<OrderItem> items;
    private Address shippingAddress;

}

The easiest approach is to serialize the entire object and publish it.

eventPublisher.publish(order);

It works—until the domain model changes. A field gets renamed, an object is restructured, or a new relationship is introduced. Every consumer now receives a different payload. The producer evolved, but the contract changed accidentally.

This is one of the most common mistakes in event-driven systems. Events should not expose internal domain models; they should communicate business facts. Those are two very different things.

An Event Describes Something That Already Happened

A useful mental model is that a command asks for something to happen, while an event states that something already happened. For example:

OrderConfirmed

This is a business fact. It cannot be rejected because it has already occurred. Similarly:

PaymentCompleted InventoryReserved ShipmentCreated CustomerRegistered

All describe completed business actions.

Consumers should be able to trust that these events represent facts. This makes event names extremely important. Good names communicate completed business outcomes, while poor names often expose implementation details. Compare the following examples:

Good: OrderConfirmed
Poor: UpdateOrderStatus

The first describes something that happened, while the second sounds like an internal method call. This distinction becomes increasingly important as systems grow.

3. Events Are Immutable

One characteristic separates events from many other forms of communication: events are immutable. Once published, an event represents history, and history cannot be rewritten.

Imagine the following event:

{
  "orderId": "ORD-1001",
  "status": "CONFIRMED"
}

Tomorrow, the customer cancels the order. The producer should not update the previous event. Instead, it publishes a new one:

{
  "orderId": "ORD-1001",
  "status": "CANCELLED"
}

The event stream now tells a complete story:

OrderCreated
      |
OrderConfirmed
      |
OrderCancelled

Consumers joining later can reconstruct what happened. This is one of the reasons event-driven systems are valuable—events become an immutable business history. Changing previously published events destroys that history.

Events Represent Facts, Not State

Another common misunderstanding is treating events as snapshots of current state. Consider this event:

{
  "orderId": "ORD-1001",
  "status": "PROCESSING"
}

Does it mean the order is currently processing, or that the order entered processing? These are different meanings.

A better event would be: OrderProcessingStarted

The name clearly communicates a business fact. Consumers no longer need to interpret the payload because the event itself explains what happened. As event catalogs grow, this principle becomes increasingly important. Well-designed event names reduce ambiguity, while poorly named events force consumers to infer business meaning.

4. Events and Commands Are Not the Same

This is one of the most misunderstood topics in event-driven architecture. Many teams use commands and events interchangeably, which creates tightly coupled systems. Understanding the difference changes how services communicate.

Commands Express Intent

A command represents a request where the sender expects another service to perform an action.

For example:
ReserveInventory

The sender is asking, “Please reserve inventory.” The receiving service can accept it, reject it, validate it, or return an error.

Commands imply responsibility—someone is expected to do something.

Events Express Facts

An event communicates that something has already happened.

For example:
InventoryReserved

The reservation has already completed. Consumers cannot reject it; they simply react. This difference may appear subtle, but architecturally it is enormous. Commands influence behavior, while events communicate history.

Visualizing the Difference

Command

Order Service
      |
Reserve Inventory
      |
      v
Inventory Service

The Order Service knows exactly who should process the request, making the communication directed.

Event

InventoryReserved
        |
   +----+----+---------+
   |         |         |
   v         v         v
Billing   Shipping   Analytics

The Inventory Service simply publishes a business fact. It does not know who reacts, and it does not need to. That is the essence of loose coupling.

A Practical Rule of Thumb

One guideline that helps during system design reviews is simple: use a command when one specific service is responsible for performing an action, and use an event when informing any interested service that a business fact has already occurred.

This distinction keeps responsibilities clear and prevents event-driven systems from gradually turning into distributed RPC systems disguised as messaging.

In the next part, we will build on these foundations by designing event schemas that survive years of system evolution.

Assisted AI to paraphrase.

JVM Internals for Microservices: Classloading, Memory, and GC in Containers

Venkatesan Ramar — Wed, 08 Jul 2026 23:34:52 +0000

A few years ago, an iPaaS platform I worked started experiencing intermittent pod restarts in Kubernetes. The issue initially appeared unrelated to the integration workloads themselves. Customer integrations were processing messages successfully, API response times remained stable, and the platform showed no obvious signs of distress. Yet several integration runtime pods were being restarted multiple times a day.

At first, the investigation focused on application-level concerns. Since the platform handled large volumes of transformation logic, message routing, and connector execution, the assumption was that some integration flow was creating excessive object allocations or causing a memory leak. Heap utilization, however, remained well below the configured limits. Garbage collection logs looked healthy, and profiling revealed no significant retention issues.

The actual problem was hidden outside the heap. The JVM was consuming memory from multiple sources that were not visible in the standard application dashboards. Metaspace grew as connectors, SDKs, and framework components loaded thousands of classes. Thread stacks accumulated because integration runtimes maintained pools for message processing, scheduling, and external system communication. Direct buffers allocated by networking libraries consumed native memory. Garbage collector metadata and JVM internal structures added additional overhead.

Individually, none of these memory consumers appeared problematic. Together, they pushed the process beyond the container's memory limit.

The JVM monitoring tools showed healthy heap usage. Kubernetes, however, cared only about the total memory consumed by the process. Once the runtime exceeded its container limit, the Linux kernel terminated it, and Kubernetes restarted the pod.

That incident highlighted an important reality of modern B2B integration platforms. Many production issues are not caused by transformation logic, connector implementations, or external system latency. Instead, they originate from misunderstandings about how the JVM behaves inside containers.

Developers building integration services often focus on workflows, mappings, APIs, and connectivity while treating the JVM as a black box. In traditional deployment environments, that approach was often sufficient. Modern cloud-native integration platforms operate under very different constraints. Memory limits are tighter, startup times affect autoscaling behavior, workloads are highly dynamic, and infrastructure efficiency directly impacts operating costs.

Modern Java runtimes have improved dramatically over the last decade. Java 21 running inside Kubernetes behaves very differently from Java 8 running on dedicated virtual machines. Nevertheless, understanding a few key JVM internals remains essential for building efficient and reliable integration services.

This article focuses on the JVM concepts that matter most in containerized environments: memory layout, classloading, and the relationship between JVM behavior and container resource limits.

1. Why JVM Internals Matter in Microservices

Many JVM concepts were easy to ignore in traditional monolithic deployments because infrastructure resources were abundant. A typical enterprise application might run as a single JVM process with access to large amounts of memory and CPU resources.

1 JVM
16 GB RAM
32 CPUs

In such environments, inefficiencies often remained hidden. An application consuming an extra few hundred megabytes of memory rarely caused operational problems. Startup times were less important because deployments happened infrequently. Thread counts could grow significantly without immediately impacting system stability.

The cloud-native world operates under a completely different set of assumptions.

50 Services
512 MB each

Instead of one large application, organizations often deploy dozens or hundreds of smaller services. Each service operates within strict resource boundaries. Every deployment has memory limits, CPU quotas, startup requirements, autoscaling behavior, and infrastructure costs associated with it.

As a result, inefficiencies that were previously insignificant become highly visible.

A service wasting 100 MB of memory may not seem problematic in isolation. However, when that same inefficiency exists across one hundred services, the organization is effectively allocating an additional 10 GB of memory simply to support overhead.

10 GB of unnecessary memory allocation

At scale, these inefficiencies translate directly into infrastructure costs, operational complexity, and reduced platform efficiency.

Cloud Cost Becomes a JVM Problem

Organizations often invest heavily in optimizing databases, networking infrastructure, and cloud architecture. Surprisingly, the JVM itself is frequently overlooked despite being one of the largest consumers of compute resources in many enterprise environments.

Consider two services that deliver identical throughput and latency. One requires 2 GB of memory while the other requires only 1 GB.

2 GB memory

versus

1 GB memory

From a business perspective, the second service is significantly more efficient. Across dozens of deployments, the difference can represent substantial infrastructure savings.

This means JVM tuning is no longer merely a performance concern. It becomes an architectural and financial concern as well.

Memory efficiency directly influences cloud spending.

Scaling Magnifies JVM Decisions

Many JVM-related decisions appear harmless during development because developers typically run a single service on a powerful workstation.

1 Service

Production environments tell a different story.

50+ Services

Every JVM incurs overhead. Every service loads classes. Every service allocates memory for threads. Every service performs garbage collection.

When multiplied across an entire platform, these costs become significant.

Classloading overhead scales.
Memory overhead scales.
Thread overhead scales.
Garbage collection overhead scales.

Understanding JVM internals becomes increasingly important as systems grow.

2. The JVM Memory Model Most Developers Never See

When developers think about JVM memory, they usually think about one thing: Heap

The heap is certainly important because it stores application objects and is the primary target of garbage collection. However, the heap represents only one portion of the JVM's memory footprint.

This distinction becomes critical in containerized environments because Kubernetes and operating systems measure total process memory consumption rather than heap usage alone.

A service can have perfectly healthy heap utilization and still be terminated due to memory pressure.

Understanding where memory is allocated inside the JVM helps explain why this happens.

Heap Memory

The heap stores the majority of application objects created during execution.

Examples include:

Order order = new Order();
List customers = ...
Map cache = ...

Whenever objects are instantiated, memory is typically allocated within the heap.

Modern garbage collectors organize the heap into multiple regions or generations. Although implementation details vary between collectors, the general concepts remain familiar.

Common concepts are:
Young Generation and Old Generation

Most objects are short-lived. They are created, used briefly, and then discarded. These objects typically remain in the young generation.

Objects that survive multiple garbage collection cycles are eventually promoted into the old generation, where they remain for longer periods.

Heap memory is usually the easiest JVM memory area to monitor because most observability tools expose heap metrics by default.

Unfortunately, this visibility often creates the misconception that heap memory represents total JVM memory consumption.

Metaspace

Starting with Java 8, Metaspace replaced the older Permanent Generation (PermGen).

Metaspace stores information about loaded classes, methods, fields, and other runtime metadata required by the JVM.

For example:
class metadata
method metadata
runtime class information

Modern Spring applications load thousands of classes during startup.

For example:
Spring Framework
Spring Boot
Hibernate
Jackson
Logging libraries
Database drivers

In addition to framework classes, many frameworks generate classes dynamically at runtime.

For instance:
Spring proxies
AOP proxies
Hibernate proxies
Bytecode-enhanced entities

Every loaded class consumes Metaspace.

Large enterprise applications can easily load several thousand classes before processing their first request. As a result, Metaspace can become a meaningful contributor to overall memory consumption.

Unlike heap memory, Metaspace often receives little attention until it becomes a problem.

Thread Stacks

Every platform thread created by the JVM receives its own stack.

Typical stack sizes range between 256 KB – 1 MB depending on operating system and JVM configuration.

This memory is allocated independently of the heap.

The impact becomes significant when applications create large numbers of threads.

Consider a service configured with 500 Threads and a stack size of 1 MB. Thread stacks alone may consume 500 MB before accounting for application objects, caches, or framework overhead.

This is one reason thread-heavy applications often consume substantially more memory than expected.

Native Memory

Native memory is one of the least understood areas of JVM memory management.

The JVM frequently allocates memory outside the heap for performance reasons.

For example:
NIO buffers
JNI libraries
Compression libraries
GC metadata
JVM internal structures

These allocations do not appear in heap metrics.

From the perspective of many monitoring dashboards, this memory is effectively invisible.

However, the operating system and Kubernetes still count it toward the process memory limit.

As a result, applications can experience memory-related failures even when heap utilization appears completely normal.

The Memory Layout That Kubernetes Sees

Kubernetes does not distinguish between heap memory, Metaspace, thread stacks, or native allocations.

It sees a single process consuming memory.

A simplified representation looks like this:

The container limit applies to the entire process, not just the heap.

This distinction explains many seemingly mysterious OOMKill incidents.

3. Classloading: The Invisible Startup Cost

Classloading is one of the most fundamental JVM mechanisms, yet most developers rarely think about it.

Every Java application depends on classloading. Every framework depends on classloading. Every object created by the application ultimately relies on classes being loaded into memory.

Despite its importance, classloading remains largely invisible during day-to-day development. Its effects, however, are highly visible.

Startup time, memory consumption, deployment speed, and auto-scaling responsiveness are all influenced by classloading behavior.

What Happens During Startup

When a Spring Boot application starts, the JVM performs a series of operations before the application becomes available.

A simplified view looks like this:

Class Loading
     |
Verification
     |
Initialization
     |
Bean Creation
     |
Application Ready

Thousands of classes may be loaded and initialized during this process. Each class must be located, verified, linked, and prepared for execution. The larger the application and dependency graph, the more work the JVM must perform before serving traffic.

Why Spring Applications Load So Many Classes

Modern Spring applications rely heavily on framework capabilities such as dependency injection, reflection, annotations, auto-configuration, and proxy generation.

These features provide tremendous developer productivity but introduce startup overhead.

Common contributors include:

dependency injection
reflection
annotations
auto-configuration
proxy generation

Even a seemingly simple dependency can trigger substantial framework activity.

For example, spring-boot-starter-web does far more than provide an embedded web server.

Spring performs extensive classpath scanning, conditional configuration evaluation, bean registration, and framework initialization.

As applications grow, startup complexity grows as well.

The Classloader Hierarchy

The JVM organizes classloading through a hierarchy of classloaders.

Most applications rely on three primary classloaders:

Bootstrap ClassLoader
          |
Platform ClassLoader
          |
Application ClassLoader

The Bootstrap ClassLoader loads core JDK classes such as java.lang.String, java.util.List

The Application ClassLoader loads application-specific dependencies and business code like:

Spring
Hibernate
Business code

Understanding this hierarchy becomes important when diagnosing dependency conflicts, startup failures, or class visibility issues. Many difficult startup problems ultimately originate from classloading behavior.

Why Classloading Matters in Containers

Classloading directly affects startup performance.

In traditional environments, startup time might not matter significantly because applications remained running for weeks or months.

Containerized environments are different.

Auto-scaling events, rolling deployments, and node failures can trigger frequent application startups.

Consider a deployment scaling from: 5 Pods to 50 Pods

Every new pod must complete startup before it can serve traffic.
Classloading delays become deployment delays.

Large applications may spend a surprising amount of time loading classes and initializing frameworks before becoming operational. For highly dynamic environments, startup efficiency becomes an important operational characteristic.

4. Memory in Containers: Where Reality Gets Interesting

Many JVM misconceptions become apparent only after applications move into containers.

Historically, the JVM was designed for physical servers and virtual machines where resource boundaries were relatively straightforward.

Containers introduced a new abstraction layer that changed how resources are allocated and enforced.

These changes exposed assumptions that were previously hidden.

The JVM Was Not Originally Container-Aware

Older JVM versions viewed the environment primarily through the perspective of the host machine.

For example:

64 GB Host

The JVM assumed those resources were available.

Containers introduced a different reality:

64 GB Host
|
512 MB Container

The application could access only a small fraction of the host's resources.

Early JVM versions frequently sized memory pools based on host capacity rather than container limits.

This behavior caused numerous production issues in Kubernetes environments.

Modern JVMs have become significantly more container-aware, but understanding the historical context helps explain many configuration recommendations.

Heap Is Not Total Memory

One of the most common mistakes in containerized Java deployments is allocating the entire container budget to the heap.

For example:

Container Limit = 512 MB
Heap = 512 MB

This configuration leaves no room for any other JVM memory consumers.

The JVM still requires memory for:

Metaspace
Thread stacks
Direct memory
Native allocations

Eventually, total process memory exceeds the container limit.

A healthier configuration might look like:

Container Limit = 512 MB

Heap = 300 MB
Remaining Memory = JVM Overhead

The exact allocation depends on workload characteristics, but the principle remains universal.

The heap cannot consume the entire memory budget.

Why Pods Get OOMKilled

Many production incidents follow a similar pattern.

Consider the following memory breakdown:

Container Limit = 512 MB

Heap = 320 MB
Metaspace = 80 MB
Native = 90 MB
Thread Stacks = 50 MB

Total = 540 MB

From the JVM's perspective,
heap utilization may appear healthy.
Garbage collection may appear healthy.
Application latency may appear healthy.

Yet, Kubernetes observes only one fact:

540 MB > 512 MB

The process exceeds its memory limit. The operating system terminates it.

Kubernetes restarts the pod.

This behavior often surprises some because traditional JVM monitoring focuses heavily on heap metrics while ignoring other memory consumers.

In many real-world incidents, understanding total JVM memory consumption provides far more value than tuning garbage collection parameters.

Before optimizing GC, it is often worth ensuring that the JVM's complete memory footprint actually fits within the container budget.

5. Garbage Collection Choices That Matter

Garbage Collection discussions often become overly theoretical. Many articles spend significant time explaining concepts such as mark-and-sweep algorithms, generational memory management, compaction strategies, and collector internals. While these topics are important for understanding how the JVM works, most backend teams are usually trying to answer more practical questions. They want to know which garbage collector they should use, what problems it solves, when it makes sense to move away from the default configuration, and how to determine whether garbage collection is actually responsible for a performance issue.

Modern Java has already made many sensible decisions on behalf of developers. For most microservices, the default collector is an excellent starting point. The real challenge is understanding when the default stops being sufficient and what trade-offs alternative collectors introduce.

What Garbage Collection Is Optimizing

Every garbage collector attempts to balance three competing goals: throughput, latency, and memory efficiency. Improving one of these dimensions often comes at the expense of another.

A collector optimized for throughput may allow longer stop-the-world pauses because it prioritizes maximizing the amount of useful application work completed over time. This approach can be highly efficient for batch workloads but may introduce noticeable pauses that affect user-facing applications.

A latency-focused collector takes a different approach. Instead of maximizing throughput, it attempts to minimize pause times by performing more work concurrently with application threads. This keeps applications responsive but consumes additional CPU resources because the collector remains active while the application is running.

Memory efficiency introduces yet another dimension. Some collectors require additional metadata structures, forwarding information, or concurrent processing overhead to achieve lower pause times. As a result, reducing latency may increase memory consumption or CPU utilization.

Conceptually, the trade-off looks like this:

                 Throughput
                      ▲
                      │
Memory Efficiency ◄───┼───► Latency

Moving closer to one corner generally means moving farther away from another. There is no universally optimal collector because every workload has different priorities. The best choice depends on the characteristics of the application being deployed.

Understanding Generational Collection

Before discussing specific collectors, it is useful to understand why modern JVMs organize memory into generations. This design is based on a simple observation: most objects in Java die young.

Objects such as HTTP request wrappers, DTOs, temporary collections, serialization buffers, and JSON parsing structures are often created and discarded within milliseconds. A single request may allocate thousands of short-lived objects that become unreachable immediately after the response is returned.

Because of this behavior, the JVM separates memory into regions optimized for different object lifetimes.

Young Generation
      |
Most Objects Die
      |
      v
Old Generation
      |
Long-Lived Objects

The young generation is collected frequently because reclaiming memory there is usually inexpensive. Objects that survive multiple collection cycles are promoted into the old generation, where they are assumed to have longer lifetimes.

Many memory-related problems begin when allocation rates become excessive, objects survive longer than expected, or old-generation growth becomes continuous. In practice, understanding object lifetime patterns is often more valuable than switching collectors because many performance issues originate from application behavior rather than collector choice.

Class Data Sharing (CDS)

Class Data Sharing is one of the least discussed JVM optimizations despite its practical value.

Normally, every JVM process loads and processes core JDK classes independently. CDS allows pre-processed class metadata to be stored in a shared archive.

JDK Classes
      |
Create Archive
      |
Shared By JVMs

This reduces startup time, lowers memory consumption, and improves class-loading efficiency. Multiple JVM processes can reuse the same archive, making CDS particularly valuable in containerized environments where many identical workloads run simultaneously.

Application CDS

Application CDS extends the same concept beyond JDK classes.

Instead of sharing only core platform classes, applications can include framework classes, third-party libraries, and application-specific code within the archive.

Application Classes
        |
Generate Archive
        |
Reuse During Startup

As application size grows, the benefits become increasingly noticeable. Large Spring Boot applications can often achieve meaningful startup improvements through Application CDS.

AOT and Native Images

Spring and GraalVM introduced another strategy: moving work from runtime to build time.

Instead of performing extensive runtime analysis, applications can be compiled ahead of time.

Build Time Analysis
        |
Generate Native Binary
        |
Fast Startup

This approach delivers faster startup times, lower memory footprints, and reduced runtime initialization overhead.

The trade-offs include longer build times, reduced JVM dynamism, reflection constraints, and additional operational complexity. Native images solve a different set of problems than traditional JVM tuning, and many organizations achieve acceptable startup performance without leaving the JVM ecosystem.

6. Observability: What To Monitor

Many JVM tuning efforts fail because teams focus on the wrong metrics. Heap utilization alone rarely tells the complete story.

Effective JVM observability requires visibility into memory behavior, allocation patterns, garbage collection activity, and runtime characteristics.

Heap Metrics

Teams should monitor heap used, heap committed, and heap maximum values. These metrics help answer important questions.

Is heap usage growing continuously? Are objects surviving longer than expected? Are caches oversized? Does memory return to normal levels after traffic decreases?

A healthy pattern often resembles:

Heap Usage
    /\
   /  \
  /    \
 /      \
----------

Memory grows and shrinks as collections occur.

A problematic pattern looks more like:

Heap Usage
    /
   /
  /
 /
/

Continuous growth may indicate memory leaks, retained references, or unbounded caches. Heap metrics are often the fastest way to identify memory-related issues.

GC Metrics

Garbage collection metrics deserve equal attention. Teams should monitor pause duration, collection frequency, allocation rate, and promotion rate.

Questions worth asking include whether pauses are affecting latency, whether allocation rates are unusually high, whether old-generation occupancy is growing continuously, and whether promotion rates are increasing unexpectedly.

Allocation rate is particularly important because a service allocating several gigabytes per second may experience GC pressure even when heap utilization appears healthy. Tuning decisions should be based on these metrics rather than assumptions.

Container Metrics

Many JVM dashboards stop at heap metrics, but container platforms don't. Teams should monitor RSS memory, container memory usage, memory limits, and OOMKill events.

Container Memory
       |
+------+------+------+
| Heap | Native | OS |
+------+------+------+

The JVM controls only part of total process memory consumption. Container-level metrics frequently expose issues that remain invisible when observing heap metrics alone.

Startup Metrics

Startup metrics are increasingly important in Kubernetes, serverless environments, and autoscaling workloads.

Useful measurements include startup duration, loaded class count, and bean initialization time.

A startup breakdown often looks like:

Startup Time
      |
      +-- Class Loading
      |
      +-- Spring Initialization
      |
      +-- Bean Creation
      |
      +-- Application Ready

Startup performance should be treated as production performance because it directly affects deployment and recovery behavior.

7. Common JVM Mistakes in Microservices

Some JVM-related mistakes appear repeatedly across engineering teams.

Setting Heap Equal To Container Memory

A common mistake is configuring heap size equal to the container memory limit.

Container Limit = 1 GB
Heap = 1 GB

This leaves no room for Metaspace, thread stacks, native memory, direct buffers, or JIT compiler structures.

Actual process memory consumption includes:

Heap + Metaspace + Native Memory + Thread Stacks

The result is often container OOMKills. A safer approach reserves sufficient memory for non-heap consumers.

Ignoring Native Memory

Many teams monitor heap usage while ignoring total process memory.

Native memory includes thread stacks, direct byte buffers, JNI allocations, GC metadata, and JIT compiler structures. These components can consume substantial memory and frequently explain crashes where heap utilization appears normal.

Premature GC Tuning

A common troubleshooting pattern looks like this:

The application slows down.
Garbage collection becomes the primary suspect.
The collector is changed.

In many cases, the actual root cause is inefficient queries, memory leaks, oversized caches, or excessive serialization overhead.

GC tuning should always be evidence-driven. Teams should validate a clear correlation between latency issues and GC pauses before changing collectors.

Latency
     |
GC Pause Correlation

Treating Every Service The Same

Different workloads have different requirements.

An API gateway is typically latency-sensitive:

API Gateway
     |
Latency Sensitive

A batch processor is usually throughput-sensitive:

Batch Job
   |
Throughput Sensitive

The gateway may benefit from lower pause times, while the batch processor may prioritize throughput and memory efficiency. JVM decisions should reflect workload characteristics rather than organizational defaults.

8. Practical Recommendations

A few practical guidelines consistently work well across many environments.

Small Spring Boot Services (< 2 GB)

For smaller services, Java 21, G1GC, and container-aware JVM defaults are usually sufficient.

Teams should monitor heap usage, RSS memory, and GC pauses while avoiding aggressive tuning. Modern JVM defaults are already highly optimized.

High-Traffic APIs

For high-traffic APIs, focus on allocation rate, latency distribution, GC pause behavior, and tail latency.

A useful investigation flow looks like:

Latency Increase
       |
Check GC Pauses
       |
Check Allocation Rate
       |
Evaluate Collector

If latency requirements justify it, ZGC may be worth evaluating.

Startup-Sensitive Services

Services that scale frequently should focus on class loading, startup configuration, CDS archives, and bean initialization performance.

Startup should be measured continuously, and regressions should be treated with the same seriousness as runtime performance regressions.

Memory-Constrained Containers

When operating in memory-constrained environments, reserve capacity for Metaspace, thread stacks, direct buffers, and other native allocations.

Container Limit
       |
       +-- Heap
       |
       +-- Metaspace
       |
       +-- Native Memory

Avoid allocating the entire container budget to the heap. Sufficient headroom is essential for stable operation.

9. Final Thoughts

Understanding JVM internals is not about memorizing garbage collector algorithms or collecting obscure JVM flags.

Microservices introduce operational constraints that make certain JVM concepts impossible to ignore. Class loading influences startup behavior. Memory extends far beyond the heap. Garbage collection affects latency, throughput, and resource utilization. Container limits apply to the entire JVM process, not just the heap.

Most production incidents involving Java services are not caused by obscure JVM bugs. They are usually the result of incorrect assumptions about how the JVM behaves inside modern cloud environments.

The JVM has become remarkably efficient. Modern collectors such as G1GC and ZGC solve problems that previously required extensive manual tuning. The responsibility of engineering teams is understanding the relatively small set of JVM concepts that directly affect production systems.

These concepts influence cloud costs, deployment speed, application stability, and operational reliability far more than any collection of JVM tuning flags ever will.

Data Consistency Under Contention: Optimistic vs Pessimistic Locking

Venkatesan Ramar — Wed, 24 Jun 2026 06:35:00 +0000

A few years ago, I investigated a production issue where customers occasionally reported incorrect inventory counts. The application was healthy. The database was healthy. No errors appeared in the logs.

The problem turned out to be concurrent updates. Multiple requests were modifying the same inventory record at nearly the same time, and one update silently overwrote another. The database did exactly what it was asked to do. The application failed to co-ordinate concurrent modifications to shared data.

This is a common consistency problem. Whenever multiple users, services, or processes attempt to modify the same data simultaneously, contention appears.

To manage that contention, systems typically rely on two approaches Optimistic locking and Pessimistic locking. Both aim to preserve data consistency, but they make very different assumptions about how conflicts occur. Those assumptions directly affect performance, scalability, and user experience.

1. Why Locking Exists

Databases are excellent at storing and retrieving data, but they do not inherently understand business intent. They execute operations exactly as instructed. This becomes problematic when multiple users interact with the same piece of data at the same time.

Consider an inventory record:

Product A
Inventory = 10

Now imagine two users accessing the system simultaneously. Both users read the same inventory value:

Inventory = 10

User A purchases one item.
User B purchases two items.

The timeline looks like this:

User A reads 10
User B reads 10

User A writes 9
User B writes 8

Both transactions succeed from the database's perspective. No errors occur, and both updates are accepted. However, one update effectively overwrites the other.

This scenario is known as a lost update. Both users started with the same information, but because their updates were not co-ordinated, one user's changes disappeared. Locking mechanisms exist primarily to prevent such situations.

Concurrency Is Usually A Business Problem

Concurrency issues rarely present themselves as obvious technical failures. Systems continue running, databases remain available, and monitoring dashboards look healthy. The real impact appears in business outcomes.

Customers do not care whether the root cause involves MVCC, transaction isolation levels, or a particular locking strategy. They only see incorrect results. For that reason, concurrency control is not merely a database concern—it is a business requirement that directly affects customer trust and operational correctness.

Contention Changes Everything

Many applications operate flawlessly until contention increases. A user profile system may rarely experience concurrent updates because different users modify different records. In contrast, a payment platform may process thousands of updates against the same accounts every second. Similarly, a seat reservation system may have thousands of users competing for a very small number of records.

The frequency of contention is one of the most important factors when choosing a concurrency strategy. Systems with frequent conflicts require a different approach than systems where conflicts are rare. This distinction forms the foundation of the optimistic versus pessimistic locking debate.

2. Pessimistic Locking: Assume Conflict Will Happen

Pessimistic locking starts with a conservative assumption:

Someone else will probably try to modify this data.

Because conflicts are expected, the system prevents them from occurring by restricting access immediately. The first transaction acquires a lock on the data, and any subsequent transaction attempting to modify the same data must wait until the lock is released.

This approach prioritizes correctness by ensuring that only one transaction can modify a resource at a time.

The Bank Account Example

Imagine two transactions attempting to modify the same account balance.

Transaction A begins and acquires a lock:

SELECT * FROM account WHERE id = 100 FOR UPDATE;

The row becomes locked, preventing other transactions from modifying it.

Now Transaction B attempts the same operation:

SELECT * FROM account WHERE id = 100 FOR UPDATE;

Because Transaction A already holds the lock, Transaction B cannot proceed. It must wait until Transaction A completes and releases the lock. This guarantees that updates occur sequentially rather than concurrently.

What Happens Under Contention

The flow looks like this:

Notice that the second transaction does not fail. Instead, it pauses until the lock becomes available. This behavior makes correctness easier to reason about because the database itself enforces exclusive access to the data. Developers do not need to detect conflicts later because the database prevents them from occurring in the first place.

Why Financial Systems Like Pessimistic Locking

Certain domains prioritize correctness above all else. Examples include payment processing systems, banking platforms, trading applications, and inventory reservation systems.

In these environments, waiting is preferable to risking inconsistent data. Consider two users attempting to reserve the last available airline seat. Allowing both requests to proceed simultaneously could result in overselling the seat, creating operational and customer-service problems. A short delay is usually a much smaller cost than correcting inconsistent business data later.

The Cost Of Waiting

While pessimistic locking provides strong protection against conflicting updates, it introduces a different challenge: reduced concurrency.

As contention increases:

response times increase
throughput decreases
blocked transactions accumulate

Under heavy load, lock contention can become a significant bottleneck. Instead of processing business operations, the database spends more time coordinating access to shared resources. This trade-off becomes increasingly visible in high-traffic systems where many users compete for the same records.

3. Optimistic Locking: Assume Conflict Is Rare

Optimistic locking takes the opposite approach.

Most transactions will not conflict.

Instead of preventing concurrent access, the system allows multiple users to work with the same data simultaneously. Rather than blocking access upfront, conflicts are detected later when an update is attempted.

This approach assumes that contention is relatively uncommon and that most operations can proceed without interference.

The Core Idea

Optimistic locking typically relies on a version number stored alongside each record.

Example:

Account
--------
Id = 100
Balance = 1000
Version = 5

Suppose two users read the same row. Both receive:

Version = 5

User A updates the record first:

UPDATE account SET balance = 900, version = 6
               WHERE id = 100 AND version = 5;

The update succeeds because the version matches the expected value. The record now becomes:

Version = 6

Later, User B attempts an update:

UPDATE account SET balance = 800, version = 6
               WHERE id = 100 AND version = 5;

This update affects zero rows because the version is no longer 5. The database detects that another transaction modified the record first, and the update fails.

Conflict Becomes Explicit

Unlike pessimistic locking, optimistic locking does not force transactions to wait. Instead, conflicting updates fail immediately.

The application must then decide how to respond. Common options include:

retry
refresh data
reject the operation
ask the user to resolve the conflict

This approach makes conflicts visible rather than hiding them behind waiting transactions. The responsibility for handling those conflicts shifts from the database to the application.

Why Modern Applications Prefer Optimistic Locking

Many business applications experience relatively low contention. Examples include customer profiles, employee records, product catalogs, and content management systems. Most users interact with different records, making simultaneous updates uncommon.

In these environments, blocking every update would introduce unnecessary overhead. Optimistic locking allows the system to maximize concurrency while still detecting the occasional conflict. As a result, applications achieve better scalability and responsiveness.

The Cost Of Retrying

Optimistic locking reduces database contention but introduces complexity elsewhere. Because conflicts are detected after they occur, applications must implement strategies for handling failures.

Retries may sound straightforward, but production systems require additional considerations such as exponential back-off,
user experience, duplicate submissions and retry storms.

As a result, conflict resolution becomes an important part of application design rather than a purely database-level concern.

4. How Modern RDBMS Actually Handle Concurrency

Many engineers imagine databases constantly locking rows and blocking transactions. Modern relational databases are far more sophisticated.

Systems such as PostgreSQL and MySQL rely heavily on a technique called Multi-Version Concurrency Control (MVCC). Understanding MVCC helps explain why modern databases can support high levels of concurrency without excessive blocking.

Multiple Versions Of Data

Instead of immediately replacing existing data, MVCC creates new versions of rows whenever updates occur.

Conceptually:

┌───────────────┐
│ Row Version 1 │
└───────────────┘
         ↓
┌───────────────┐
│ Row Version 2 │
└───────────────┘
         ↓
┌───────────────┐
│ Row Version 3 │
└───────────────┘

Older versions remain available for active transactions that still need them. This allows readers to continue accessing a consistent view of the data while updates occur in parallel.

The result is significantly less blocking and much higher concurrency.

Why Reads Usually Don't Block Writes

One of the most common misconceptions about databases is:

Every update blocks every read.

In MVCC-based databases, this is not true. Readers can access a consistent snapshot of the data while writers create newer versions in the background.

This capability allows databases to support large numbers of concurrent users without forcing readers and writers to constantly wait for one another. It is one of the primary reasons modern relational databases scale far better than many developers initially expect.

Isolation Levels Matter

Locking strategies are only one part of the consistency story. Isolation levels determine what data a transaction can see while other transactions are running.

Common isolation levels include:

Read Committed
Repeatable Read
Serializable

Each level provides different guarantees and trade-offs. Higher isolation levels generally offer stronger consistency but require additional coordination and overhead.

Choosing a locking strategy without understanding transaction isolation can lead to incorrect assumptions about application behavior. In practice, consistency emerges from the combination of locking mechanisms, MVCC behavior, and transaction isolation working together.

5. Deadlocks: The Hidden Cost of Pessimistic Locking

Pessimistic locking guarantees exclusive access to data by preventing multiple transactions from modifying the same resource simultaneously. While this approach is highly effective at preserving consistency, it introduces a different class of concurrency problems: deadlocks.

Deadlocks typically do not appear during initial development or testing because contention levels are low and transaction flows are relatively simple. As systems grow, however, more users, background processes, and business workflows begin interacting with the same data concurrently. Under these conditions, transactions may start waiting on each other in ways that create circular dependencies.

When that happens, transactions that previously completed successfully begin failing unexpectedly, without any changes to the underlying business logic.

A Classic Deadlock Scenario

Consider a money transfer workflow involving two accounts.

Transaction A                     Transaction B
─────────────                     ─────────────
Lock Account A                    Lock Account B
       │                                 │
       ▼                                 ▼
Update Account A                  Update Account B
       │                                 │
       ▼                                 ▼
Lock Account B ◄──────────────► Lock Account A
                  DEADLOCK

Transaction A holds a lock on Account A and waits for Account B. Meanwhile, Transaction B holds a lock on Account B and waits for Account A.

Neither transaction can proceed because each will be waiting for a resource currently held by the other. Neither transaction can release its lock because it has not yet completed.

The database detects this circular wait condition and identifies it as a deadlock.

Deadlocks are not limited to two rows or two transactions. In complex systems, deadlocks may involve multiple tables, indexes, and transactions, making them difficult to diagnose without proper monitoring and logging.

How Databases Resolve Deadlocks

Modern relational databases continuously analyze lock dependencies between active transactions. When a deadlock is detected, the database must break the cycle to allow progress.

A simplified flow looks like this:

┌───────────────┐
│ Transaction A │
└───────────────┘
         ↓
┌───────────────┐
│   Deadlock    │
└───────────────┘
         ↓
┌──────────────────────────┐
│ Database Chooses Victim  │
└──────────────────────────┘
         ↓
┌───────────────┐
│   Rollback    │
└───────────────┘

The database selects one transaction as the deadlock victim and rolls it back. The other transaction is allowed to continue and eventually commit.

The victim selection process varies by database implementation. Factors such as transaction age, resource consumption, and rollback cost may influence which transaction is terminated.

From the application's perspective, this usually appears as an exception indicating that the transaction failed due to a deadlock. The application must be prepared to retry the operation because deadlocks are considered transient failures rather than permanent errors.

Importantly, deadlocks are not database bugs. They are an expected consequence of concurrent transactions acquiring locks in different orders.

Deadlocks Become Operational Problems

Deadlocks are difficult to reproduce in development environments because concurrency levels are significantly lower than in production.

Real-world systems contain many independent actors operating simultaneously like concurrent users, background jobs, asynchronous consumers and scheduled tasks.

Each of these components may access shared resources using different execution paths.

A deadlock occurring once every few weeks may have little operational impact. However, when contention increases and deadlocks begin occurring hundreds or thousands of times per hour, they can significantly affect throughput, latency, and user experience.

For this reason, high-scale systems attempt to minimize lock durations, enforce consistent lock acquisition ordering, or adopt optimistic concurrency strategies when contention remains relatively low.

6. Optimistic Locking in Spring and JPA

Optimistic locking is one of the most commonly used concurrency control mechanisms in enterprise Java applications. Frameworks such as JPA and Hibernate provide built-in support, making implementation straightforward while still offering strong protection against lost updates.

Unlike pessimistic locking, optimistic locking does not prevent concurrent access. Instead, it detects whether another transaction modified the data between the time it was read and the time it was updated.

The @Version Annotation

A typical entity might look like:

@Entity
public class Account {

    @Id
    private Long id;

    private BigDecimal balance;

    @Version
    private Long version;
}

The @Version field acts as a concurrency token. Every successful update increments the version number automatically.

When Hibernate generates update statements, it includes the current version value in the WHERE clause. This ensures that updates only succeed if the record has not been modified since it was originally read.

This mechanism allows multiple users to read the same data concurrently while still preventing silent overwrites.

What Actually Happens

Suppose two users load the same entity.

Both receive:
Version = 10

User A updates first.

The version becomes:
Version = 11

User B attempts an update.

Hibernate generates an update statement similar to:

UPDATE account SET balance = ?, version = 11
               WHERE id = ? AND version = 10

Because the row now contains version 11 instead of version 10, the WHERE condition no longer matches.

As a result, no rows are updated.

Hibernate detects this condition and throws OptimisticLockException. This exception indicates that another transaction modified the entity after it was originally loaded.

Rather than silently overwriting data, the application is forced to acknowledge and handle the conflict.

Handling Optimistic Lock Failures

Adding @Version annotation is only the first step.

The more important challenge is deciding how the application should respond when conflicts occur.

Possible strategies include:

retry automatically
reject the operation
reload and merge
notify the user

The appropriate choice depends heavily on business requirements.

For example, inventory/reservation systems retry automatically because conflicts are expected and transient. Collaborative editing systems may present users with merge options. Financial applications frequently reload the latest state and re-validate business rules before attempting another update.

Optimistic locking provides conflict detection. It does not provide conflict resolution. Designing an effective resolution strategy is a critical part of building reliable systems.

7. Locking in NoSQL Databases

A common misconception is that NoSQL databases eliminate concurrency concerns.

In reality, concurrent modification problems still exist. The difference lies in how databases expose consistency guarantees and concurrency control mechanisms.

Most NoSQL platforms provide some form of optimistic concurrency control rather than traditional row-level locking.

MongoDB

MongoDB provides atomic operations at the document level. Updates to a single document are isolated and executed atomically.

For concurrency control, many applications implement version-based optimistic locking.

Example:

db.orders.updateOne(
  {
     _id: 1,
     version: 5
  },
  {
     $set: {
        status: "SHIPPED"
     },
     $inc: {
        version: 1
     }
  }
)

The update succeeds only if the document still contains version 5.

If another process updates the document first, the query condition no longer matches:

Matched Documents = 0

The application can then detect the conflict and decide whether to retry or reject the operation.

Conceptually, this is very similar to optimistic locking in relational databases.

Redis

Redis is generally viewed as a simple in-memory cache, but it is also frequently used as a primary data store, coordination mechanism, and distributed locking platform.

Because Redis executes commands sequentially within a single-threaded event loop, individual commands are atomic. However, concurrency challenges still arise when multiple clients perform read-modify-write operations.

One approach is to use optimistic concurrency control through the WATCH command.

Example:

WATCH account:100
GET account:100

The client reads the value and prepares an update.

When the transaction executes:

MULTI
SET account:100 900
EXEC

Redis verifies that the watched key has not changed since it was read.

If another client modifies the key before EXEC, the transaction is aborted: Transaction Failed

The application can then retry using the latest value.

Redis is also widely used for distributed locking through commands such as:

SET resource-lock unique-id NX PX 30000

This creates a lock only if the key does not already exist and automatically expires it after a specified timeout.

While distributed locks can co-ordinate access across multiple application instances, they should be used carefully. Improper lock expiration settings, network partitions, and process failures can introduce subtle consistency issues.

For this reason, many Redis-based systems prefer optimistic concurrency patterns or idempotent operations whenever possible, reserving distributed locks for workflows that truly require exclusive access.

DynamoDB

DynamoDB provides optimistic concurrency control through conditional writes.

A write operation can specify a condition that must evaluate to true before the update is applied.

The following example performs an UpdateItem operation. It tries to reduce the Price of a product by 75—but the condition expression prevents the update if the current Price is less than or equal to 500.

aws dynamodb update-item \
    --table-name ProductCatalog \
    --key '{"Id": {"N": "456"}}' \
    --update-expression "SET Price = Price - 75" \
    --condition-expression "Price > 500"

If the starting Price is 650, the UpdateItem operation reduces the Price to 575. If you run the UpdateItem operation again, the Price is reduced to 500. If you run it a third time, the condition expression evaluates to false, and the update fails.

This approach allows DynamoDB to maintain high scalability while still preventing lost updates. Because conditional writes are implemented directly by the storage engine, applications can enforce concurrency guarantees without introducing explicit locking mechanisms.

Many large-scale AWS systems rely heavily on this pattern.

8. Distributed Systems Change Everything

Many engineers discover an uncomfortable reality when transitioning from monolithic applications to microservices:

Database locking does not extend beyond a single database.

Traditional locking mechanisms work extremely well within a single transactional boundary. Once data and business processes span multiple services, those guarantees disappear.

Locks Cannot Cross Services

Consider:

Order Service
      |
Database A

and

Inventory Service
      |
Database B

A lock acquired in Database A has no effect on Database B.

Even if both services participate in the same business workflow, neither database has visibility into the other's locks or transactions.

As a result, traditional database locking cannot guarantee consistency across service boundaries.

This limitation fundamentally changes how distributed systems are designed.

Why SAGAs Exist

Microservices frequently execute workflows that span multiple services and databases.

Example:

Create Order
      │
      ▼
Reserve Inventory
      │
      ▼
Process Payment
      │
      ▼
Create Shipment

No single ACID transaction can encompass the entire workflow.

Instead, systems rely on:

compensating transactions
retries
eventual consistency

This is the problem Saga patterns address.

Rather than locking resources across services, Sagas coordinate a sequence of local transactions and define recovery actions when failures occur.

The goal is not immediate consistency but reliable business outcomes despite partial failures.

Why Outbox Doesn't Require Locks

The Transactional Outbox pattern solves a different challenge.

It guarantees Database Commit + Event Publication without requiring distributed transactions.

The application writes both business data and an outbound event record within the same local transaction. A separate process later publishes the event.

This approach relies on transactional guarantees within a single database.

Not pessimistic locking.

Understanding this distinction is important because many distributed systems problems are fundamentally reliability and coordination problems rather than concurrency-control problems.

Idempotency Beats Locking

Many distributed systems avoid locking altogether.

Instead, they make operations idempotent, meaning the same operation can be executed multiple times without changing the final outcome.

Example:

Process Payment Event

The consumer records Payment Already Processed and ignores duplicates.

This strategy allows systems to safely retry operations without introducing global locks or distributed coordination.

Modern event-driven architectures frequently prefer:

retries
idempotency
eventual consistency

over distributed locking because these approaches scale more effectively and remain resilient during failures.

9. Choosing Between Optimistic and Pessimistic Locking

Neither optimistic nor pessimistic locking is universally superior.

The correct choice depends on workload characteristics, contention frequency, consistency requirements, and performance goals.

Understanding how often conflicts occur is usually more important than understanding the locking mechanism itself.

Choose Pessimistic Locking When

Pessimistic locking is most appropriate when conflicts are common and the cost of inconsistency is high.

Scenarios like:

seat reservation systems
inventory allocation
financial transactions
account balance updates

In these scenarios, allowing concurrent modifications may create unacceptable business outcomes. Waiting for access is preferable to resolving conflicts after they occur.

Correctness takes priority over throughput.

Choose Optimistic Locking When

Optimistic locking works best when conflicts are relatively rare.

Scenarios like:

customer profiles
product catalogs
employee records
content management systems

Most transactions complete successfully without interference from other users. Because contention is low, avoiding locks improves concurrency and reduces database overhead.

The occasional conflict can be handled through retries or user intervention.

Measure Contention First

Many teams choose a locking strategy based on assumptions rather than evidence.

A better approach is to measure:

lock wait time
retry rates
update conflicts
transaction latency

Production metrics reveal surprising patterns.

A workflow that appears highly contentious may rarely experience conflicts, while seemingly independent operations may compete heavily for shared resources.

Data should drive concurrency decisions whenever possible.

10. Common Mistakes Teams Make

Concurrency control is generally misunderstood because systems behave correctly under low load and fail only when contention increases.

Several mistakes appear repeatedly across production systems.

Using Pessimistic Locking Everywhere

Applying pessimistic locking indiscriminately can severely limit scalability.

The application remains correct, but:

throughput decreases
latency increases
lock contention grows

As traffic increases, the database spends more time coordinating access than executing business logic.

Correctness is essential, but excessive locking can become a significant performance bottleneck.

Ignoring Retry Logic

Optimistic locking assumes conflicts will occasionally occur. Without retry mechanisms, users may experience unnecessary failures even when a simple retry would succeed immediately.

Applications should treat optimistic lock exceptions as expected outcomes rather than exceptional situations. Proper retry policies are as important as the locking strategy itself.

Long Transactions

Locks held for extended periods dramatically increase contention. Transactions should perform only the work necessary to maintain consistency.

External API calls, file processing, and lengthy computations should generally occur outside transactional boundaries whenever possible.

Short transactions reduce lock duration and improve overall system throughput.

Confusing Isolation Levels with Locking

Many developers assume Serializable automatically solves every concurrency problem.

In reality, isolation levels define visibility rules between transactions, while locking strategies define how concurrent modifications are co-ordinated.

Both influence consistency.
Neither replaces the other.

Understanding the distinction is critical when diagnosing concurrency issues.

11. Final Thoughts

Concurrency control is fundamentally the discipline of managing contention while preserving correctness.

Optimistic and pessimistic locking approach this challenge from different perspectives.

The correct choice depends on:

contention patterns
consistency requirements
throughput goals
operational behavior

Many production systems use both approaches simultaneously. Critical workflows may require strict exclusivity, while less contentious operations benefit from maximum concurrency.

The most effective engineers understand the trade-offs behind each strategy and apply them deliberately based on business requirements and real-world traffic patterns. Because concurrency problems rarely appear when systems are idle. They appear when traffic grows, users increase, and contention finally arrives.

Assisted AI to generate charts and diagrams.

Build vs Buy: The Expensive Engineering Decision Less Talked About

Venkatesan Ramar — Mon, 15 Jun 2026 08:59:00 +0000

Back in 2015, I joined a product company whose platform had been evolving since late 90's. Coming from a startup background, I was overwhelmed by the number of in-house tools, and platforms that existed alongside the core product.

Over time and after leaving the organization in 2023 — I began to appreciate the trade-offs behind those build decisions. Some became strategic assets, while some introduced years of ownership and maintenance overhead.

This article shares some of the lessons I learned about one of the most important engineering decisions teams make: build or buy.

Over the years, I've come to believe that some of the most expensive engineering mistakes have very little to do with technology itself.

They start with a much simpler question:

Should we build this ourselves?
Or should we buy it?

At first glance, the answer often feels obvious. A team identifies a need.
Maybe it's:

authentication,
workflow orchestration,
internal developer portals,
database migration tool, or
some internal framework.

Someone says:

"We can build this in a few weeks."

Often, they're right. The first version usually isn't that difficult. The real challenge comes later. Because every build decision eventually becomes an ownership decision.

And ownership tends to last much longer than implementation.

Over the years, I've seen teams successfully build internal platforms that became strategic assets. I've also seen teams accidentally become software vendors to themselves.

The interesting question isn't whether we can build something. Modern engineering teams can build almost anything.

The more important question is:

Do we want to own it for the next five years?

1. Why This Decision Matters

A decade ago, many engineering teams had fewer choices. But today, the situation is completely different.

Almost every technical capability has mature products available.

Say, you need:

authentication?
observability?
workflow orchestration?
developer portals?

There is probably a vendor already solving that problem, that's what makes the decision difficult.

Because modern engineering teams are no longer choosing between having a capability, or not having one.

They're choosing between:

building it,
extending it,
buying it, or
integrating it.

The number of options has increased. At the same time, engineering capacity remains limited. Teams hit decision paralysis.

Every sprint spent building internal tooling is a sprint not spent building customer-facing capabilities.

This trade-off becomes increasingly important as organizations grow. Especially when platform investments start competing with product investments.

2. The Hidden Cost Teams Ignore

One pattern I've noticed is that teams are usually good at estimating development effort. They're much less effective at estimating ownership effort.

A discussion might sound like this:

This looks straightforward. We can probably build it in three weeks.

Probably they're right. The problem is that the three-week estimate usually covers only Version 1.

It rarely includes:

upgrades,
support,
operational maintenance,
bug fixes,
security reviews,
documentation,
on-boarding,
scalability improvements, and
future requirements.

Those costs appear gradually which makes them easy to underestimate.

Building Is Easy. Owning Is Hard

Many internal systems begin life as small engineering utilities. Gradually adoption grows. Soon other teams depend on them.

Now expectations change.

The platform suddenly needs:

up-time guarantees,
backward compatibility,
support processes, and
clear ownership.

What started as an engineering project slowly becomes a product except now the customers are internal teams.

I've seen this happen with:

internal frameworks,
workflow engines,
authentication services, and
developer portals.

The implementation wasn't the difficult part but the long-term ownership was.

The Internal SaaS Trap

One of the most interesting things about platform engineering is that organizations sometimes become software vendors without realizing it.

Imagine a team builds an internal feature flag platform.

Version 1.0 supports simple enable/disable toggles. Seems pretty straightforward.

Then adopted teams raise feature requests like percentage roll-outs, audit logs, experimentation, approval workflows.

Now the platform team is effectively running a software product. Except instead of external customers, they're supporting internal engineering teams.

The complexity didn't disappear. It simply became your responsibility.

Opportunity Cost Is Real

This is probably the most overlooked factor in build-versus-buy discussions.

Suppose five engineers spend six months building an internal platform.

The direct cost is obvious but another question is often ignored:

What didn't get built during those six months?

Perhaps:

product features were delayed,
customer requests remained unresolved,
roadmap commitments slipped.

These costs rarely appear in dashboards. Yet they often have a larger business impact than infrastructure costs.

3. Why Engineering Teams Choose To Build

Despite the risks, engineering teams continue to build internal solutions. There are good reasons for that, not every build decision is a mistake.

Some become enormous competitive advantages.

The challenge is understanding why we are choosing to build in the first place.

Engineers Like Solving Problems

This shouldn't surprise anyone.

Most engineers enjoy creating systems, and building software feels productive and empowering. It provides a level of control that third-party products cannot. When requirements are unique, building can absolutely make sense.

The problem appears when technical enthusiasm replaces strategic evaluation.

Just because something is technically interesting doesn't automatically mean it should be owned long-term.

Vendor Skepticism

Many teams have legitimate concerns about vendors.

Questions such as:

What if pricing changes?
What if the company gets acquired?
What if we're locked in?
What if customization becomes difficult?

These concerns are real. Sometimes they justify building.

But I've also seen teams dramatically overestimate vendor risks while underestimating ownership risks.

Both sides deserve equal scrutiny.

The "It Looks Simple" Fallacy

Some capabilities appear deceptively simple.

Authentication is a classic example.

At first glance:

users log in,
users log out,
passwords are stored.

Simple, until requirements expand:

OAuth
SAML
MFA
SSO
compliance
account recovery
security reviews

Suddenly the original problem looks very different.

Version 1 is usually easy. Version 10 is where the complexity appears.

4. Where Companies Successfully Build

It might sound like buying is always the safer option, No. Many of the most successful technology companies built substantial internal platforms.

The difference is that they usually built capabilities closely tied to their competitive advantage.

Build What Makes You Different

One repeated pattern among successful engineering organizations is that they build things that are core to their business. Not things that are merely useful.

This distinction matters.

Netflix Didn't Win By Building Authentication

Netflix became successful because of:

streaming infrastructure,
recommendation systems,
content delivery,
personalization.

Those capabilities directly influenced the business.

Investing heavily in them made strategic sense. Authentication was necessary. Recommendation systems were differentiating.

The difference is important.

Uber Didn't Buy Dispatching

Dispatching is central to Uber's business.

The way drivers and riders are matched directly affects customer experience, efficiency, profitability.

That capability is core business logic.

Owning it provides competitive advantage. Buying it would have limited differentiation.

LinkedIn Built Kafka

Kafka began as an internal project at LinkedIn to solve large-scale event streaming challenges. At the time, existing messaging systems struggled to handle the volume, durability, and scalability requirements of LinkedIn's growing platform.

Building Kafka made sense because reliable event streaming was becoming a foundational capability for the business. What started as an internal solution eventually evolved into one of the most widely adopted distributed systems in the industry.

The lesson isn't that every company should build its own messaging platform. The lesson is that LinkedIn built a capability that directly addressed a strategic problem at its scale.

Google Built Kubernetes

Kubernetes originated from Google's experience running massive distributed systems over many years. Google had already developed internal container orchestration platforms and operational practices long before containers became mainstream.

Rather than adapting existing solutions, Google built Kubernetes based on lessons learned from operating infrastructure at enormous scale.

For most organizations, building a container orchestration platform would be a terrible investment. For Google, infrastructure management was a core competency and strategic advantage.

The takeaway is simple:

Build when the capability is closely tied to your unique scale, business model, or competitive advantage.

The Common Pattern

The most successful build decisions usually share a characteristic:

The capability directly contributes to competitive advantage.

When that's true, ownership often makes sense. When it doesn't, the equation changes dramatically.

5. The Platform Engineering Perspective

Over the last few years, platform engineering has become a major focus area for many organizations. The goal is to make developers more productive.

Provide:

self-service capabilities,
deployment automation,
observability,
infrastructure provisioning, and
standardized workflows.

The challenge is deciding how much of that platform should be built internally.

This is where build-versus-buy decisions become particularly interesting.

The Internal Developer Platform Dilemma

Imagine a growing engineering organization.

Developers complain about:

inconsistent environments,
deployment complexity,
on-boarding difficulties.

The organization decides to build an Internal Developer Platform. The initial vision sounds reasonable.

A central portal where developers can create services, access documentation, provision resources, monitor deployments. But soon new requirements emerge:

RBAC
audit logs
integrations
workflow automation
plugin ecosystems

Before long, the platform itself becomes a product.

The Backstage Lesson

Many organizations faced this exact challenge.

Instead of building an entire developer portal from scratch, they adopted existing platforms and customized them.

This approach is interesting because it reflects a broader engineering principle:

Buy the foundation. Build the differentiation.

The organization still owns the developer experience. It avoids spending years recreating foundational capabilities.

Build The Last 20%

One of the most useful heuristics I've encountered is this:

Buy the first 80%. Build the last 20%.

The first 80% usually consists of commodity functionality. The last 20% often contains:

business-specific workflows,
domain integrations,
unique operational requirements.

That final layer is often where competitive advantage exists. It's usually a better place to invest engineering effort.

6. A Practical Decision Framework

Over time, I've found that build-versus-buy discussions become much easier when evaluated through a consistent framework.

Rather than debating technologies, the conversation shifts toward business and ownership.

Here are some of the questions help to make decision.

Question 1: Does This Differentiate The Business?

This is one of the most important question.

If the capability disappeared tomorrow, would customers notice?
Would it impact the company's competitive position?

For example:

Building recommendation engines, pricing algorithms, matching engines, or domain-specific workflows often creates differentiation.

The closer a capability is to competitive advantage, the stronger the case for building.

Question 2: Do We Want To Own This In Three Years?

Most build decisions focus on implementation.
Few focus on ownership.

A better question is:

Will we still want to maintain this three years from now?

Ownership includes:

upgrades,
security,
operational support,
bug fixes,
documentation,
compliance,
training.

If the answer feels uncomfortable, that is valuable information.

Question 3: Can We Support It Operationally?

Every system eventually enters production and production changes everything.

A build decision also means committing to on-call support, incident response, monitoring, maintenance, and/or disaster recovery.

The engineering effort doesn't end when the code is deployed. In many cases, that's where the real work begins.

Question 4: Is The Market Mature?

Sometimes buying is difficult because the market is immature. The available products may not solve the problem adequately.

But in mature categories observability, authentication, feature management and workflow orchestration vendors have often spent years refining their solutions.

Ignoring that accumulated expertise can be expensive.

Question 5: What Is The Opportunity Cost?

This question is frequently overlooked.

Suppose a team spends six engineers, six months, building an internal capability.

The direct cost is obvious but opportunity cost is harder to measure.

What customer-facing work was delayed?
What revenue-generating features were postponed?
What strategic initiatives slowed down?

Sometimes the most expensive cost is the one that never appears in a budget report.

7. The Hybrid Model Usually Wins

One thing I've noticed is that the engineering organizations rarely choose a pure build or pure buy strategy.

Instead, they combine both.

Buy The Foundation

Commodity capabilities are often purchased or adopted. These tools solve common problems that many organizations face.

Rebuilding them rarely creates differentiation.

Build Business-Specific Layers

The organization's engineering effort is then focused on:

business workflows,
domain models,
operational processes,
customer-facing capabilities,
proprietary integrations.

This is where engineering investment usually generates the highest return.

Why This Works

The hybrid model captures the advantages of both approaches.

Organizations avoid:

rebuilding mature capabilities,
unnecessary ownership burden,
platform reinvention.

At the same time, they retain flexibility where it matters most.

8. Common Mistakes Teams Make

Most build-versus-buy failures follow surprisingly similar patterns. The technology might change but the mistakes rarely do.

Underestimating Maintenance

Teams usually estimate 'Initial Build Cost' while forgetting support, upgrades, security and operations. Over a multi-year horizon, ownership often exceeds implementation cost.

Rebuilding Commodity Software

This is probably the most common mistake.

Engineering teams are highly capable. Given enough time, they can rebuild almost anything. The question is whether they should.

Optimizing For Engineering Preference

This is a subtle trap.

Engineers naturally enjoy building systems. But engineering satisfaction and business value are not always aligned.

A technically elegant solution can still be a poor investment. The best technical decision is not always the best business decision.

Assuming Vendors Never Improve

Many build decisions are based on current vendor limitations but products evolve. Markets mature gradually and capabilities improve. A solution that looked inadequate two years ago may look very different today.

Periodic re-evaluation is important.

9. AI Is Changing The Economics

It's impossible to discuss build-versus-buy decisions today without mentioning AI. AI-assisted development has significantly reduced implementation effort. Many teams can now prototype internal tools faster than ever.

Capabilities that once required months of development can sometimes be assembled in days.

Building Is Cheaper Than Before

AI helps with:

scaffolding
code generation
testing
documentation
integration

The barrier to building has unquestionably decreased.

Ownership Has Not Become Cheaper

This is the important distinction.

AI can help create software. It does not eliminate:

operational support
on-call responsibility
compliance
security
upgrades
platform ownership

The cost of creation is decreasing. The cost of ownership remains surprisingly stable. That means ownership becomes even more important in future build-versus-buy discussions.

10. Final Thoughts

One of the biggest lessons I've learned is that build-versus-buy decisions are rarely technology decisions.

They're ownership decisions.

Modern engineering teams can build almost anything. Open-source ecosystems are thriving. Cloud platforms provide powerful building blocks.

AI accelerates development even further.

The question is no longer: Can we build it?
The more important question is: Do we want to own it?

Because every build decision creates a long-term commitment. A commitment to maintenance, operations, support, upgrades, and continuous evolution.

Sometimes that commitment is absolutely worth making especially when the capability creates competitive advantage. Other times, the smarter decision is to leverage what already exists and focus engineering effort where it matters most.

In the end, the most successful engineering organizations aren't the ones that build everything. They're the ones that understand what is truly worth owning.

Assisted ChatGPT to rephrase.