Replication gives us another copy of the truth. Then the network disappears, and both copies have to decide whether they are still allowed to call themselves true.
"A practical guide to the CAP theorem, exploring consistency, availability, network partitions, split brain, quorum, and why distributed systems must choose what correctness means when communication fails."
Imagine a hotel booking system running on one server.
Bob asks for Room 12.
The server checks.
Room 12 is available.
Bob gets the room.
There is something ✨beautifully✨ uncomplicated about this. One server owns the state, so there is one obvious answer to the question:
Is Room 12 available?
But one server is also one failure away from having no booking system at all.
So we do the sensible thing.
Add another server.
Perhaps one serves one region and another serves somewhere else. Keep their booking records synchronized. If one machine becomes unavailable, the other can continue serving customers.
More redundancy.
More availability.
More confidence.
Until Bob asks Server A for Room 12 at exactly the wrong moment, while Dong asks Server B for the same room.
Normally, the servers coordinate.
This time, the connection between them has disappeared.
Server A checks what it knows.
Available.
Server B checks what it knows.
Also available.
Bob gets Room 12.
Dong gets Room 12.
Both servers behaved perfectly according to the reality they could see.
And together they produced an impossible hotel.
That is the uncomfortable beginning of CAP.
The Problem Is Not That Either Server Is Wrong
At first, it is tempting to blame synchronization.
Surely the servers should simply synchronize before confirming the booking.
That does solve the problem while communication is working.
But now keep the network partition.
Server A cannot reach Server B.
What should A do when Bob asks for the room?
There are two obvious choices.
It can say:
I cannot verify the current state, so I refuse to complete the request.
The data remains safe from contradictory bookings.
But Bob has encountered an unavailable service even though Server A itself is alive and perfectly capable of computing.
Or Server A can say:
I cannot reach B, but according to my local state the room is free, so I will continue.
Bob gets a response.
The service remains available.
But Server B may independently tell Dong exactly the same thing.
The system cannot guarantee both behaviours simultaneously.
That tension is what the CAP theorem formalizes.
In the usual formulation:
- Consistency means clients observe one coherent, up-to-date view of the data.
- Availability means requests to non-failing parts of the system receive responses.
- Partition tolerance means the system must cope with communication being lost between parts of the network.
Gilbert and Lynch formally showed that in an asynchronous network subject to partitions, a distributed service cannot guarantee both strong consistency and availability during such a partition.
The network has forced a decision.
Not because the architecture was badly implemented.
Because the two servers no longer have enough information to make both promises.
“Pick Two” Is a Little Too Comfortable
CAP is often introduced as a triangle:
Consistency. Availability. Partition Tolerance.
Pick two.
It is memorable.
It is also dangerously easy to interpret as a shopping list.
Want consistency and availability? Choose CA.
Want availability and partition tolerance? Choose AP.
Want consistency and partition tolerance? Choose CP.
The diagram makes these choices look wonderfully symmetrical.
Reality is less cooperative.
In an actual distributed system, we do not control whether a cable is cut, a switch fails, packets disappear, or two healthy machines temporarily lose their ability to communicate.
We do not get to configure: network.partition = falseand move on with our day.
AWS’s current CAP guidance makes the practical implication explicit: distributed systems generally have to tolerate network failures, so when a partition occurs, the meaningful trade-off becomes consistency versus availability.
That changes the question.
It is no longer:
Which two letters would you like?
It becomes:
When communication breaks, what is the system allowed to sacrifice?
That is a much less decorative question.
And a much more useful one.
CP Says “I Would Rather Refuse Than Lie”
Go back to Room 12.
Server A cannot communicate with Server B.
If we care strongly about consistency, Server A cannot confidently approve Bob’s reservation.
Perhaps B has already sold the room.
A does not know.
That uncertainty matters.
So a CP-style design may refuse operations that cannot be completed safely while the partition exists.
This can feel frustrating.
Both servers are alive.
The database process is running.
The CPU is fine.
The request arrived.
And the system still says no.
But the refusal is doing real work.
The system is protecting an invariant:
One room must not be successfully reserved twice.
Consistency becomes more important than allowing every surviving partition to continue independently.
This same idea appears in distributed locking and majority-based protocols. A node may be technically alive yet deliberately prevented from acting because it lacks enough agreement from the rest of the system.
The service has not forgotten how to process the request.
It has lost the authority to process it safely.
That distinction matters.
AP Says “I Would Rather Answer and Reconcile Later”
Now imagine a different kind of data.
You edit a file while disconnected from the network.
Should the application freeze until every other copy of that file is reachable?
That would certainly make synchronization easier.
It would also be a fairly cursed user experience.
So an AP-style system can make the opposite trade.
Let the reachable side continue operating.
Accept that different replicas may temporarily disagree.
Then reconcile them later.
This leads naturally toward weaker consistency guarantees, including eventual consistency.
The distinction is useful:
Strong consistency says that after an update completes, subsequent accesses should observe that updated state.
Weak consistency does not guarantee that an immediate subsequent access will see it.
Eventual consistency adds an important promise: if updates stop, replicas will eventually converge toward the latest state.
Dropbox gives a wonderfully tangible version of the idea. Files marked available offline can be opened and edited without network access. Those changes do not synchronize to other devices while the machine is disconnected; synchronization resumes when connectivity returns. If the same file is changed independently, conflicts may need to be represented and resolved.
The local copy is allowed to keep living its life.
For a while, two devices may disagree.
That does not automatically mean the system has failed.
It means the system decided that temporary inconsistency was preferable to making the file unusable whenever synchronization became impossible.
AP Is Not “No Consistency”, and CP Is Not “No Availability”
This is where the simple CAP triangle can really mess with the mental model.
An AP system does not necessarily throw consistency into a bin and wish everyone luck.
A CP system does not necessarily collapse into permanent unavailability whenever one packet goes missing.
These properties have degrees, time windows, scope, and application-specific meaning.
An available system might return slightly stale information while aggressively propagating updates and resolving conflicts.
A consistency-focused system might remain completely available whenever communication is healthy and only restrict a subset of operations during an actual partition.
The trade-off becomes interesting precisely because it is not always static.
Consider an airline reservation system.
Suppose a flight has 200 seats and only 40 have been sold.
A temporary inconsistency in the exact remaining count may be tolerable. Refusing every reservation merely because one replica cannot be reached could cause more harm than allowing another booking.
Availability has room to breathe.
Now imagine 199 seats are gone.
Suddenly the remaining state matters a lot.
Two independent servers both believing that one seat remains can sell seat 200 twice.
The invariant has become tight.
Consistency now deserves more authority.
Same system.
Same data.
Different moment.
Different acceptable trade-off.
The deeper lesson is not “choose AP” or “choose CP.”
It is to understand which guarantees matter for which operations, and under which conditions.
Then the Backup Decides the Primary Is Dead
CAP becomes even easier to feel when we stop talking about database reads and watch a system accidentally create two authorities.
Consider a primary server and a backup.
Clients normally use the primary.
The primary sends updates to the backup so the backup can take over if necessary.
This is sensible redundancy.
Now break the communication link between primary and backup.
The backup stops hearing from the primary.
What should it conclude?
The primary might have crashed.
Or the network between them might simply be broken.
From the backup’s side, those situations can look painfully similar.
Suppose the failover rule says:
If the primary disappears, become the new primary.
The backup promotes itself.
Excellent.
Except the original primary never crashed.
It is still alive.
Some clients can still reach it.
Other clients can reach the newly promoted backup.
The old primary thinks:
The backup is unreachable. I am still primary.
The backup thinks:
The primary is dead. I am now primary.
Both conclusions are locally reasonable.
Globally, they are disastrous.
The system has developed a split brain.
Oracle’s clustering documentation describes the same failure pattern: after communication between cluster nodes is lost, separate partitions can each believe they are the valid cluster and continue operating independently.
Our redundancy has produced two sources of authority.
Both Halves Can Be Correct and the Whole Can Still Be Wrong
Split brain is nasty because neither side needs to be malfunctioning internally.
Imagine clients on the left continuing to write state through the original primary.
Clients on the right write different state through the promoted backup.
For a while, each half can look healthy.
Requests arrive.
Responses go out.
Data changes.
Nobody is necessarily throwing errors.
That is what makes the condition dangerous.
The problem is not that the machines stopped working.
The problem is that agreement about who is allowed to speak for the system disappeared.
Suppose the service answers a question whose correctness depends on information from both sides.
Each half can respond based on incomplete state.
Each response may be perfectly sensible from the local view.
Together, those answers can violate the assumptions of the system above them.
Split brain is CAP wearing a name tag.
The partition created separate realities.
Availability allowed those realities to continue making progress.
Consistency became impossible unless one reality stopped making independent decisions.
And now we can finally see what the solution really needs.
Not better optimism.
Not another timeout.
Agreement.
Two Machines Cannot Resolve the Argument by Staring at Each Other
The obvious fix sounds easy:
Let the primary and backup decide which one is really alive.
Unfortunately, that is almost the problem we already had.
If they can communicate reliably enough to make that decision together, they are not experiencing the partition we are trying to survive.
During the partition:
Primary cannot see backup.
Backup cannot see primary.
Each one knows only:
I am alive, and the other side is not answering.
That does not reveal whether the other side is dead.
Separate one-to-one decisions can therefore produce separate answers.
So the system needs something beyond the two conflicting participants.
One option is brutally simple:
Use one server.
If it fails, wait.
Consistency becomes straightforward, but availability during the failure suffers.
Another option is fencing: when a backup takes over, ensure the old primary is physically or logically prevented from continuing to serve.
If only one side is capable of acting, there cannot be two primaries.
And another option is majority agreement.
Now the partition must prove that it represents enough of the system before it is allowed to continue.
This is where quorum enters the story.
Majority Does Something Clever With Authority
Suppose there are enough voting members that one side of a partition contains 60 votes and another contains 40.
Only the side with more than half is allowed to continue as the authoritative cluster.
The other side may be alive.
It may have healthy disks.
It may have clients begging for service.
It still stops.
Why?
Because two disjoint groups cannot both possess a strict majority of the same electorate.
That mathematical detail is doing the heavy lifting.
It prevents two partitions from simultaneously claiming legitimate authority.
Oracle Solaris Cluster uses this exact idea: a partition with a majority of votes obtains quorum and is allowed to operate, while partitions without quorum cannot continue as the cluster. The mechanism exists specifically to prevent split-brain operation.
Notice what we paid for that guarantee.
The minority side became unavailable.
That is not an accidental flaw in the quorum mechanism.
That is the trade-off.
We deliberately sacrifice some availability because allowing both sides to continue would risk divergent state.
The architecture stops looking pessimistic once the constraint is visible.
A majority vote is not there because distributed-systems engineers enjoy making healthy servers shut themselves down.
It is there because being alive is not the same thing as being authoritative.
Partitions Do Not Break Reality. They Break Shared Reality.
This is the part of CAP that feels more useful than memorizing the triangle.
Before the partition, replicas can coordinate enough to present a common state.
Then communication disappears.
Each side continues observing its own local reality.
From that moment, the system cannot magically know what is happening beyond the partition boundary.
If both sides continue accepting conflicting updates, availability survives but immediate consistency cannot.
If the system refuses operations until sufficient agreement exists, consistency survives but some requests cannot proceed.
If users are allowed to work offline, their state may diverge temporarily and reconcile later.
If two servers both decide they are primary, redundancy becomes split brain.
If quorum grants authority to only one partition, the system stays coherent by deliberately silencing the other.
Same underlying problem.
Different manifestations.
The network did not merely disconnect machines.
It disconnected knowledge.
And once knowledge is separated, perfect consistency and perfect availability cannot both be promised as though nothing happened.
The Choice Was Never Really About Three Letters
CAP is memorable because of C, A, and P.
But the useful engineering lesson sits underneath them.
Distributed systems create copies of state because copies are useful.
They improve scalability.
They improve resilience.
They put computation and data closer to where they are needed.
But every copy introduces another participant that must somehow remain aligned with the others.
As long as communication works, synchronization can hide much of that complexity.
A partition removes that luxury.
Now the design has to reveal what it really values.
Can stale state be tolerated?
Can conflicting updates be reconciled later?
Is there an invariant that must never be violated?
Can an operation safely stop?
Who has authority during disagreement?
How is that authority established?
What happens when communication returns?
Those questions are far more useful than asking whether a database has an AP or CP sticker on the box.
The hotel booking system did not fail because we added a second server.
The backup did not become dangerous because redundancy was foolish.
The offline file was not broken because two copies temporarily disagreed.
Each design encountered the same underlying fact:
Independent machines cannot maintain one shared reality without communication, and communication itself is one of the things a distributed system cannot guarantee.
So the architecture adapts.
Sometimes it waits.
Sometimes it refuses.
Sometimes it keeps going with stale state.
Sometimes it reconciles.
Sometimes it asks a majority who is still allowed to speak.
Sometimes it checks with The Origami Software Engineer.
The network split reality in two.
The system’s job was never to pretend that did not happen.
It was to decide what correctness means when it does.
That’s not failure.
That’s evolution.
The “I liked this” Starter Pack:
Don’t let your fingers get lazy now.
- Like : It tells me this was worth writing.
- A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
- Boost it: Especially with that one developer who definitely needs this.
Thanks for being here. It genuinely helps more than you know!
— Aaroophan Varatharajan
Find me elsewhere:
- Professional stuff: linkedin.com/in/Aaroophan
- Code stuff: github.com/Aaroophan
- UI stuff: aaroophan.dev/Aaroophan
- Life stuff: instagram.com/Aaroophan
Top comments (0)