DEV Community: Aaroophan Varatharajan

The Distributed File System That Wins Depends on What You’re Afraid Of

Aaroophan Varatharajan — Fri, 09 Oct 2026 07:21:01 +0000

Why NFS, AFS, Coda, and GFS Make Different Trade-Offs for Different Failures

A technical comparison of Sun NFS, AFS, Coda, and GFS, examining how transparency, caching, scalability, availability, replication, consistency, reliability, and security constraints shaped each distributed file system.

At first, this looks like a comparison problem.

Put Sun NFS, AFS, Coda, and GFS in four columns.

Add transparency, performance, scalability, availability, reliability, security.

Give each system a score.

Highest total ✨wins✨

Perfectly reasonable.

And almost immediately wrong.

The problem is that these systems are not four contestants solving the same problem with different levels of competence. They entered the room carrying different fears.

NFS is worried about remote files feeling remote.

AFS is worried about thousands of clients exhausting the servers.

Coda is worried about the network simply disappearing.

GFS assumes machines will fail routinely and asks how to keep massive data processing moving anyway.

So the useful question is not: Which distributed file system is best?

It is: Best under which constraint?

Transparency Looks Easy Until the Network Starts Having Opinions

The first instinct is simple: if a file is remote, hide that fact.

Sun NFS takes this intuition seriously. Its protocol is explicitly designed for transparent remote access , with RPC carrying operations across the network and XDR providing a common representation across different machines, operating systems, network architectures, and transport protocols.

NFS is the diplomat here. It wants everybody to speak file system without caring too much about where the bytes live.

Its stateless server follows the same philosophy. The server avoids maintaining protocol state about its clients, which makes recovery after failure dramatically simpler: clients can retry operations instead of requiring the server to reconstruct an elaborate shared history.

That feels beautifully clean.

Then scale arrives.

AFS wants the same kind of location-transparent experience, but it expects thousands of workstations. If every read keeps dragging Vice, the trusted server side, back into the conversation, Vice eventually becomes the colleague who is somehow invited to every meeting.

So AFS gives Venus, the client cache manager, more responsibility.

Venus caches entire files locally. Reads and writes can then happen against the cached copy, while Vice keeps the responsibilities that actually require server authority. The system preserves a homogeneous, location-transparent namespace while deliberately pushing work toward clients.

The simple mental model was:

transparency means hiding the network.

AFS reveals the deeper model:

transparency also depends on deciding who must participate in each operation .

That distinction becomes surprisingly expensive later.

Mobility Exposes the Assumption AFS Was Still Making

AFS caching solves a lot.

But it quietly assumes that the server is still reachable when the client actually needs it.

That assumption feels harmless until a portable workstation leaves the network.

Now the cached copy is sitting there, perfectly usable, while the system technically has nobody authoritative to talk to.

The obvious conservative answer is to stop.

No server, no update.

Safe. Consistent. Miserable.

Coda refuses that bargain.

It inherits much of the AFS model but introduces two complementary mechanisms: server replication and disconnected operation. During disconnected operation, a caching site temporarily assumes the role of a replication site, allowing useful work to continue even when the shared repository is inaccessible. The design explicitly identifies portable workstations as a major use case.

This is where the architecture gets uncomfortable.

If a disconnected client can modify data, another client may modify its own copy elsewhere.

Now both copies are sincere.

Both have history.

Both think they are right.

Coda chooses an optimistic strategy because preventing all such divergence would destroy much of the availability it is trying to create. Conflict detection and later reconciliation become the price of continued operation.

That is the hidden constraint:

Mobility is not just remote access from another machine. It is maintaining useful authority when the normal authority is unreachable.

Once that requirement appears, “just stay consistent” stops being a complete answer.

Performance Stops Being a Number

It is tempting to ask which system is fastest.

Fast at what?

GFS makes this question impossible to ignore.

Its world contains large files, streaming reads, append-heavy writes, commodity machines, many clients, and applications that care more about high sustained bandwidth than low latency.

Under those assumptions, traditional file-system instincts start looking suspicious.

GFS uses a single master for metadata and many chunkservers for data. Clients ask the master where chunks live, then communicate directly with chunkservers for the actual data path. The master knows where everything is, but it very deliberately refuses to carry everybody’s luggage.

Large chunks reduce metadata overhead and repeated master interaction.

Replication assumes failures will happen.

Record append reflects the actual workload instead of pretending random overwrite is equally important.

This is specialization without apology.

AFS reaches performance from the opposite direction.

Its enemy is server interaction.

Whole-file transfer and client caching mean Venus can satisfy much of the workload locally. Callback-based cache validation further reduces the need for clients to repeatedly ask servers whether cached data is still valid. Howard and colleagues identify whole-file transfer and caching as central to AFS’s ability to scale gracefully.

So GFS says:

make the data path bypass central metadata control.

AFS says:

make repeated data access bypass the server entirely.

NFS stays more conservative. Its stateless RPC model gives it simplicity and portability, but that same simplicity keeps the client-server relationship comparatively chatty. Even the protocol specification acknowledges that server simplicity can become awkward when clients want richer file-system semantics.

Nobody is “faster.”

They are removing different bottlenecks.

Scalability Is Usually Someone Becoming Tired

Scale has a habit of revealing whichever component everyone politely assumed would cope.

In AFS, that component is Vice.

So Venus takes work away from it.

In GFS, that component could be the master.

So clients cache metadata and transfer data directly with chunkservers.

In NFS, statelessness keeps servers conceptually simple, but repeated remote interactions remain part of the model.

In Coda, scale is complicated further because availability mechanisms themselves create state: replicas must be tracked, cached versions must be reconciled, and disconnected clients eventually have to return carrying edits made in another reality.

This is why scalability is not just “add more machines.”

Adding machines adds:

more metadata,
more coordination,
more replicas,
more failure combinations,
more opportunities for stale state,
more people asking the same exhausted authority for answers.

The architecture scales when it decides which conversations no longer need to happen.

Availability and Reliability Look Similar Until Something Actually Fails

Coda and GFS both care deeply about failure, but their definitions of surviving it are different.

Coda asks: Can the user keep working?

Server replication protects against some server and network failures. Disconnected operation covers the much uglier case where no appropriate server can be reached at all.

GFS asks: Can the storage service keep functioning while components routinely die?

Its answer includes chunk replication, monitoring, master recovery mechanisms, stale-replica detection, and checksums. Google explicitly designed GFS around the assumption that component failures are normal in clusters built from inexpensive commodity hardware.

Same word: availability.

Different anxiety.

And reliability gets even more specific.

A replica existing somewhere is not enough.

What if it missed an update?

What if its disk corrupted the data?

What if a client reads from something that looks alive but belongs to yesterday?

GFS uses chunk version numbers to identify stale replicas and checksums to detect corrupted data. Replication is therefore not just duplication; replicas must remain trustworthy enough to participate. The verification work around GFS further examines its relaxed and eventual consistency behaviour, showing that consistency here is an explicit system property rather than a vague promise.

The replica does not get authority merely because it still has a pulse.

That is a useful distinction.

Data Integrity Reveals Why Availability Has a Price

The seductive model of replication is: make three copies and relax.

Distributed systems do not relax.

GFS coordinates mutations across replicas and uses versioning and checksumming to identify stale or damaged copies. Its relaxed consistency model is not equivalent to abandoning integrity; instead, the system defines what applications may observe and shapes the interface around workloads that can tolerate those guarantees.

Coda makes a different trade.

It wants updates to remain possible across disconnection and partitions.

That means temporary divergence can be legitimate.

The system therefore needs mechanisms for detecting conflicts and reconciling them later.

This can feel like integrity has been weakened.

A better way to see it is that the definition of integrity changed because the availability requirement changed.

GFS tries hard to keep replicas participating in one coordinated storage reality.

Coda sometimes permits two realities to exist temporarily because refusing the second reality would mean refusing the user.

Neither choice is free.

That is the part the tidy architecture diagrams usually leave out.

Security Is Where Old Trust Assumptions Start Showing

NFS began with portability, transparency, and simple recovery as dominant concerns.

Its classic design consequently places considerable trust in the surrounding authentication environment. The protocol’s early security model relies on mechanisms such as AUTH_UNIX, while the stateless server must repeatedly reason about permissions without maintaining rich client state.

That trade becomes increasingly uncomfortable as the environment gets less friendly.

AFS and Coda operate with a more explicit trusted-server model, and Coda’s design includes authentication and encrypted communication because mobility, caching, and replication make misplaced trust especially expensive.

GFS is different again. Its foundational design discussion spends enormous attention on failure tolerance, replication, throughput, and consistency, but adversarial multi-tenant security is not one of its central design axes.

That does not automatically mean “insecure.”

It means something more important when comparing architectures: absence of emphasis is not evidence of strength.

A system deserves credit for the problem it actually solves, not for requirements we wish it had discussed.

The Leaderboard Was the Wrong Abstraction

Once all these constraints are visible, the rankings become almost embarrassingly contextual.

Sun NFS is compelling when the problem is transparent remote access across heterogeneous systems and the architecture should stay comparatively simple. Its RPC/XDR model and statelessness follow directly from that goal.

AFS becomes compelling when the problem is large shared-workstation populations and servers cannot remain involved in every access. Whole-file caching, Venus, Vice, callbacks, and volumes are consequences of scale.

Coda becomes compelling when connectivity itself cannot be trusted. Server replication and disconnected operation follow from the requirement that users continue working through failures and mobility.

GFS becomes compelling when the environment is large, data-intensive, append-heavy, and built from hardware expected to fail. Large chunks, a metadata master, direct client-to-chunkserver communication, replication, checksums, and relaxed semantics all follow from that workload.

That is the deeper pattern.

Architecture starts looking strange when we stare only at the solution.

Once we reconstruct the constraint that came before it, the weird parts become almost inevitable.

The stateless server was not minimalism for vibes.

Whole-file caching was not an optimization somebody added because caches are fashionable.

Disconnected operation was not a feature bolted onto AFS.

GFS did not relax traditional semantics because correctness became unfashionable.

Each system moved because its original mental model eventually met a requirement it could no longer ignore.

That is what mature architecture means to The Origami Software Engineer:

Not a collection of universally correct choices,

but a record of which assumptions survived contact with the constraints.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

When "Just Make the Interface Easy" Stops Being Enough: Rethinking Human Computer Interaction

Aaroophan Varatharajan — Sun, 04 Oct 2026 14:18:39 +0000

Why goals, tasks, translation, ergonomics, interaction styles, error recovery, context, and perceived value shape usable systems

A technical exploration of Human-Computer Interaction covering Norman’s execution and evaluation gulfs, Abowd and Beale’s interaction framework, ergonomics, WIMP, interaction styles, error recovery, context, fluidity, and user value.

Interaction looks simple from a distance.

A user wants something. A system can do something. Put an interface between them, make the controls obvious, and the problem should mostly disappear.

That intuition is ✨reasonable✨

It is also where the trouble starts.

The deeper you follow interaction, the less it looks like a screen-design problem and the more it looks like two very different worlds trying to understand each other without exhausting, confusing, or misleading either side. (HCI Book)

The Seductive Model: User In, System Out

At first, interaction is simply communication between user and system.

That gives us a wonderfully clean picture: the user says something, the computer responds, and everybody goes home happy.

Except communication is not the same as understanding.

The interaction already carries four kinds of pressure: interaction models must translate between user and system, ergonomics must deal with physical characteristics, interaction styles determine the nature of the dialogue, and context brings social, organizational, and motivational forces into the room.

So the interface is not merely passing messages.

It is trying to keep two different realities synchronized.

Domain, Goal, and Task: The First Crack

The first useful separation is deceptively small.

A domain is the area of work under study.
A goal is what the user wants to achieve.
A task is how the user goes about achieving it through operations or actions.

Suppose the domain is graphic design and the goal is to create a solid red triangle. Selecting the fill tool and clicking the triangle is merely the task.

That distinction matters because interfaces are very good at making us obsess over tasks.

Buttons. Menus. Clicks. Steps.

But the user did not arrive because they wanted to click a fill tool.

They wanted the triangle.

Once that becomes clear, the interface stops being the purpose and starts becoming what it always was: the negotiator.

Norman Gives the Negotiator a Workflow

Donald Norman makes the interaction much more precise.

The user establishes a goal, formulates an intention, specifies actions at the interface, executes them, perceives the resulting system state, interprets that state, and evaluates it against the original goal.

This creates the execution/evaluation loop.

And it feels reassuringly orderly.

The user knows what they want. Their intention becomes an action. The system responds. The user observes the response and decides whether the goal was achieved.

This should work.

It does work.

Except when the user and system disagree about what any of those steps mean.

The Gulfs: Where a Correct System Can Still Feel Wrong

Now the hidden constraint becomes visible.

The Gulf of Execution appears when the actions the user formulates are not the actions the system allows.

The Gulf of Evaluation appears when the state the user expects is not how the system actually presents that state.

Nothing necessarily crashed.

Nothing necessarily malfunctioned.

The system may be completely correct internally while feeling almost hostile externally.

Then human error divides again.

A slip happens when the user understands the system and the goal, formulates the correct action, but performs the wrong action.

A mistake can begin earlier: the user may not even have the correct goal.

That distinction quietly changes responsibility.

A slip suggests better interface design.

A mistake suggests better understanding of the system.

"User error" was never one thing.

Abowd and Beale Reveal the Translation Problem

The next model makes the tension harder to ignore.

Abowd and Beale introduce four participants: User, Input, System, Output.

Each has its own language.

The user’s intentions must become actions at the interface. Those actions must become alterations of system state. That state must become output. The output must then become something the user can interpret.

Interaction is therefore translation between languages.

And the uncomfortable conclusion follows naturally:

Problems in interaction = Problems in translation.

The system is not necessarily broken.

The translation chain is.

That is why apparently small interface decisions can create surprisingly large misunderstandings. (HCI Book)

Ergonomics Reminds Software That Humans Have Bodies

It is tempting to keep solving this as a language problem.

Then ergonomics walks in carrying chairs, lighting, noise, physical position, button placement, colour, temperature, and human fatigue.

Suddenly the abstraction has a body.

Controls may need grouping according to function or frequency of use. Seating may need to accommodate different users. Colour choices need awareness of colour-blindness. The physical environment changes what is usable.

The difference between office and industrial interfaces makes this wonderfully obvious.

Office data may be textual, slow-changing, and used in a clean environment.

Industrial data may be numeric, fast-changing, and surrounded by dirt.

Then comes the perfect little reality check: The oil soaked mouse.

Nothing is technically wrong with the mouse.

The context simply made our "obvious" solution cursed.

Glass Interfaces Solve One Problem and Introduce Another

Replacing traditional dials and knobs with screens and keypads feels like progress.

A glass interface can be cheaper, more flexible, provide precise values, and show multiple representations of the same information.

Very seductive.

But physical controls knew where they lived.

The glass interface may lose physical location and context while becoming more complex.

So the solution is not always replacement.

Sometimes the design needs both.

That becomes even more important with indirect manipulation. In an office interface, the user often manipulates an artificial world directly. In an industrial interface, the user may manipulate the real world through the interface.

Now feedback and delays matter.

The interface may say the command was accepted while the physical process is still catching up.

The screen and reality are no longer the same clock.

Interaction Styles Are Different Kinds of Conversation

Once interaction is understood as dialogue, the next question becomes obvious: What kind of conversation should this be?

The command line interface gives direct access and works beautifully for repetitive tasks and expert users, but asks the user to remember its vocabulary.

Menus make options visible and replace some recall with recognition.

Natural language feels wonderfully familiar until vagueness and ambiguity arrive. The practical retreat is often to understand a subset or pick out key words.

Question/answer interfaces guide novices but restrict functionality.

Query languages such as SQL offer power, but demand understanding of structure and syntax.

Form-fills borrow familiarity from paper forms.

Spreadsheets extend that idea while maintaining consistency between values and formulas.

None of these styles is simply "better."

Each changes who carries the cognitive load.

WIMP Makes Interaction Easy, Then Becomes an Ecosystem

Then comes WIMP : Windows, Icons, Menus, Pointers.

Point-and-click interaction reduces typing. Windows divide the display into independent areas. Icons represent objects or actions. Pointers let the user select things. Menus expose available operations.

It feels almost effortless.

Then the family expands.

Buttons invoke actions. Radio buttons represent mutually exclusive choices. Check boxes support non-exclusive choices. Toolbars provide fast access to common actions. Palettes keep actions available when menus disappear. Dialogue boxes request information and then, ideally, go away.

Each component exists because another interaction created friction.

Menus save recall but consume screen space, so pop-up menus appear only when needed.

Toolbars provide speed but can become rows of mysterious icons whispering, "…but what do they do?"

Palettes exist because sometimes the menu is not there when you need it.

The interface keeps evolving because every simplification has a boundary. (HCI Book)

"Look" Was Never Enough

This is where interactivity starts to matter.

It is easy to focus on look.

The harder question is feel.

Two WIMP systems may contain windows, icons, menus, pointers, and buttons while behaving differently.

Hence: Appearance + Behaviour = Look and Feel

Speech-driven interfaces make the same lesson obvious in another form.

Speech recognition may misunderstand the user. A robust dialogue can therefore reflect its interpretation back:

"You want a ticket from New York to Boston?"

Now the user can correct it.

The system is not pretending to be perfect.

It is making misunderstanding repairable.

Initiative, Error, and the Right to Recover

Another hidden question appears: Who has the initiative?

Traditional question/answer interaction often gives it to the computer.

WIMP usually gives it to the user.

Then a modal dialog box appears and refuses to leave until the user responds.

Sometimes that is appropriate for errors or essential steps.

Use it everywhere and the interface becomes the colleague who keeps blocking the doorway to ask one more "quick question".

Errors themselves cannot always be avoided either.

So the design shifts from impossible perfection to error and repair.

Make the error easy to detect.

Then make repair possible.

That is a much more humane model of interaction because it accepts something the earlier mental model tried to hide:

Humans and systems will occasionally misunderstand each other.

Good interaction survives the misunderstanding.

Context Changes the Same Interface Without Touching the Code

By now it is tempting to believe we finally have the whole picture.

Then context widens it again.

Interaction is affected by other people, competition, desire to impress, fear of failure, allegiance, ambition, self-satisfaction, and organizational pressures.

An inadequate system can create frustration and reduce motivation.

The software may not have changed at all.

The interaction has.

That is the uncomfortable part: usability does not live entirely inside the interface.

The surrounding social and organizational environment can change how people behave with exactly the same system. (HCI Book)

"Can Use" Becomes "Want to Use"

For work systems, making interaction possible may sometimes be enough.

For home, entertainment, and shopping, another condition appears.

People must want to use it.

Now the conversation shifts toward experience, engagement and fun.

Experience can sit between anxiety and boredom.

Education may involve things a learner can just accomplish with help.

The interaction is no longer judged only by whether the task can technically be completed.

The user’s willingness to remain in the interaction becomes part of the design.

Physical Design Turns Certainty Into Trade-offs

Then physical design brings constraints that refuse to cooperate.

Ergonomic constraints say controls need sufficient size.

Physical constraints say there may not be enough room.

Safety constraints may favour one position for adults and another for children.

Aesthetic constraints want the design to look good.

Economic constraints quietly ask who is paying for all this.

There is no magic arrangement where every constraint wins.

There are design trade-offs.

The gas-cooker example makes this painfully clear: front controls may be safer and easier for adults, while rear controls may be safer for children.

The problem is not finding the "correct" position.

The problem is understanding which constraint each position satisfies.

Fluidity Is When Physical and Logical State Stop Arguing

That leads naturally to fluidity.

Do external physical aspects reflect logical effect?

Does the physical state reveal the logical state?

Do inverse actions create inverse effects?

Spring-back controls show why this matters.

They can support large selection sets efficiently, but they introduce hidden state because the control returns physically while the logical state may remain elsewhere.

The control looks innocent.

The state has moved on.

Now the user has to remember what the control no longer reveals.

A compliant interaction avoids that kind of drift where possible: state remains evident, and controls can reflect what both user and machine are doing.

Value Was the Constraint Hiding Under Everything

After all the models, controls, translations, ergonomics, styles, errors, and trade-offs, the final condition is almost embarrassingly simple.

People use something only if it has perceived value and that value exceeds cost.

Value might be getting work done, having fun, or doing something good for others.

Cost might be money, download time, or learning effort.

Then discounted future makes the equation nastier.

People heavily discount future value and future cost.

That helps explain resistance to learning: The learning effort is happening now, while the promised benefit lives somewhere in the future wearing a tiny "trust me" badge.

Hence the design pressure toward: Low barriers and high perceived present value.

Organizations face the same issue through coercion, enculturation, and emergence : force behaviour, build shared values, or design the process so individual value becomes organizational value.

Different mechanisms.

Same underlying problem.

The Interface Was Never the Real Goal

The general lesson sounds almost too simple after everything required to reach it:

If you want someone to do something, make it easy for them .

And understand their values .

"Easy" now means much more than fewer clicks.

The goal must survive translation into action.

The available actions must fit the user’s intention.

The system state must be perceivable and interpretable.

Errors must be repairable.

Physical controls must fit their environment.

Interaction styles must fit their users.

Constraints must be traded consciously.

The eventual value must justify the cost of learning and acting.

The Origami Software Engineer must keep all of this aligned.

That is why interaction design keeps growing beyond the interface.

Every time the model becomes more realistic, another hidden assumption becomes visible.

That’s not failure.

That’s evolution.

The "I liked this" Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

“Send” Is Not One Operation: Rethinking Distributed Computing #5

Aaroophan Varatharajan — Sat, 26 Sep 2026 13:24:58 +0000

The dangerous part of `send(message)` is how much meaning we quietly pack into the word “send.”

"Why send(message) is not one operation. Explore transient vs. persistent communication, synchronous vs. asynchronous messaging, queues, publish/subscribe, message brokers, MPI, collective communication, delivery guarantees, ordering, and the contracts hidden behind distributed communication."

Two processes need to communicate.

One has some data.

The other needs it.

So we do the obvious thing: send(message)

There. ✨Sent✨

Except distributed systems have a habit of taking perfectly ordinary verbs and asking uncomfortable follow-up questions.

Where is the message now?

Is it still in the sender’s memory?

Has the operating system accepted it?

Has it crossed the network?

Has the receiving machine stored it?

Has the receiving process seen it?

Has the receiving process processed it?

Can the sender disappear now?

Can the receiver be offline?

And when send() returns, which one of those things are we actually claiming happened?

That is the trap.

“Send” sounds like one operation because human communication lets us get away with vague semantics.

Distributed communication cannot.

The system eventually has to define exactly what survives, who waits, where the message lives, and what counts as success.

And once we start answering those questions, send(message) turns out to be several very different contracts wearing the same verb.

First Question: How Long Should the Message Stay Alive?

Suppose Alice sends a message to Bob.

Bob is not running.

What should happen?

One perfectly reasonable answer is:

Then the message cannot be delivered.

This is transient communication.

The communication system keeps the message only while the communicating applications and supporting communication infrastructure are active enough to handle it. If the receiver needs to be present when the message arrives, absence matters.

Transport-level communication often has this flavour.

The sender and receiver are participating in the same live conversation.

That works beautifully when both sides are expected to exist at roughly the same time.

Then consider email.

Alice sends a message at 2 p.m.

Bob opens his mail at 9 p.m.

Nobody finds this surprising.

We quietly introduced a completely different contract.

The communication system takes responsibility for preserving the message until Bob can receive it.

That is persistent communication.

The sender does not have to remain running after submitting the message.

The receiver does not need to be running when the sender submits it.

Somewhere between them, the infrastructure remembers.

The distinction is not really about messaging syntax.

It is about who owns the message while the receiver is absent.

Transient communication says:

I will carry this while the conversation exists.

Persistent communication says:

Give it to me. I will keep it until the other side can deal with it.

Same word.

Very different promise.

Second Question: When Is the Sender Allowed to Continue?

Now suppose Bob is available.

Alice sends the message.

Can Alice immediately continue doing other work?

That gives us another axis entirely.

With asynchronous communication, the sender submits the message for transmission and continues.

The message may still be sitting in a local buffer.

It may be inside an intermediate communication server.

It may not be anywhere near Bob yet.

Alice has handed off responsibility and moved on.

With synchronous communication, Alice waits.

But even this turns out not to have one obvious meaning.

What exactly is she waiting for?

Perhaps she waits until the message reaches a buffer on Bob’s machine.

That is receipt-based synchronization.

Or perhaps she waits until the message is actually delivered to Bob’s application.

That is stronger.

Or perhaps she waits until Bob processes the message and sends something back.

That is response-based synchronization.

Now the innocent verb send has become a ladder of guarantees.

Submitted.

Buffered.

Received by host.

Delivered to application.

Processed.

Responded to.

Each level tells Alice more.

Each level can also make Alice wait longer.

And this is the part that is easy to miss:

Persistent vs. Transient and Synchronous vs. Asynchronous are not the same question.

One asks:

How long can the communication system preserve the message?

The other asks:

How long must the sender wait?

Those two choices can be combined.

A message can be persistent and asynchronous, like the intuitive model of email.

Persistent and synchronous communication can wait until durable storage exists at the receiving side without requiring the receiving application itself to be running.

Transient asynchronous communication can let the sender continue immediately while relying only on temporary buffering.

Transient synchronous communication can wait for receipt, delivery, or even a response while still assuming the live communication context exists.

We did not make messaging complicated for fun.

We discovered that “sent” was hiding multiple independent decisions.

The Queue Says, “I’ll Hold It”

Once persistence matters, a useful character enters the story.

The message queue.

Instead of requiring Alice and Bob to meet directly, Alice puts a message into a queue.

Bob retrieves it later.

The queue sits between them like a patient intermediary saying:

You two do not actually need to coordinate your calendars.

Applications communicate by placing messages into queues and removing them when ready.

That produces loosely coupled communication.

Alice does not necessarily need Bob to be active.

Bob does not necessarily need Alice to still exist.

Neither side has to wait for the other to reach the exact same point in its lifecycle.

AWS describes SQS in essentially this role: a durable queue that helps decouple distributed components, allowing producers and consumers to operate without requiring them to be concurrently available.

That is a profound change in ownership.

In direct transient communication, Alice and Bob are responsible for meeting each other.

With persistent messaging, the queue accepts part of that responsibility.

And because it does, the interface itself changes.

Instead of one vague send, queue systems expose operations with much clearer meanings.

Put appends a message.

Get waits until a message exists and removes it.

Poll checks without waiting.

Notify arranges for a handler to be called when a message becomes available.

The verbs are more precise because the state of the communication is now explicit.

The message can exist independently of both applications.

Queued Does Not Mean Processed

This creates another seductive assumption.

If Alice successfully puts the message in the queue, perhaps the job is done.

Not quite.

What has actually happened?

The queue accepted responsibility for the message.

That is valuable.

It is not the same thing as Bob processing it.

Bob may not read the queue immediately.

Bob may fail while processing.

The message may remain waiting.

A queue can therefore give the sender confidence about handoff without automatically giving confidence about application completion.

Modern queue systems make this distinction visible. In Amazon SQS, for example, a message can remain stored until a consumer retrieves it; while being processed it can remain hidden for a visibility period, and the consumer deletes it only after successful processing. (docs.aws.amazon.com)

That lifecycle exists because these are different events:

The communication system has the message.

and:

The application has completed the work represented by the message.

Again, “sent” was too vague.

The queue forces us to become honest.

Then We Stop Naming the Receiver

Queues decouple sender and receiver in time.

But Alice may still be sending work toward a particular destination.

What if she does not know who should receive the information?

Suppose Alice produces an event:

Order 781 was completed.

Several systems may care.

Billing.

Analytics.

Inventory.

Notifications.

Perhaps systems we have not even built yet.

Making Alice explicitly call every consumer creates another dependency problem.

So we change the communication model again.

Alice becomes a publisher.

She publishes the event, usually to a topic.

Interested systems become subscribers.

The publish/subscribe infrastructure determines who should receive which messages.

Now communication has been decoupled in several dimensions.

The publisher and subscriber do not necessarily need to be online at the same time.

They do not necessarily need to know each other’s addresses.

And the publisher does not need to stop and wait for every subscriber to finish processing the event.

AWS’s SNS documentation describes this fan-out model directly: publishers send through a topic, which distributes messages to multiple subscribers; combined with queues, messages can also be persisted for later asynchronous processing.

The publisher says:

This happened.

Not:

Bob, please execute method X.

That is a different relationship entirely.

The components are becoming less aware of one another.

Which is excellent for loose coupling.

It also means the communication infrastructure is taking on more responsibility.

Somebody Still Has to Understand the Message

There is another hidden assumption.

Alice publishes:

{ customer: 17, total: 4500 }

Bob expects:

{ customerId: 17, amount: 4500 }

The message arrived perfectly.

Congratulations. Nobody understands each other.

Transport success does not imply semantic compatibility.

Applications still need agreed message formats, schemas, commands, and meanings.

One option is standardization.

Make every participant agree on the same format.

That works until integrating independently developed systems becomes painful.

Another option is a message broker.

The broker sits between applications and can transform incoming messages into representations understood by destination applications.

Now the intermediary is no longer merely storing and forwarding.

It may be translating.

Routing.

Filtering.

Mediating between different expectations.

The system has bought more decoupling by moving more responsibility into the communication layer.

There is always a bill somewhere.

A Message Bus Is Not a Conveyor Belt

Once many applications communicate through shared infrastructure, we can go further and define a message bus.

Applications agree on message schemas.

They agree on command messages.

They use shared infrastructure for sending those messages to recipients.

This feels pleasantly orderly.

Then comes another assumption worth losing:

If I sent A and then B, surely everyone receives A and then B.

Not necessarily.

Routing can change.

Messages can be buffered.

Internal optimization can alter paths.

Different transports may behave differently.

Some messaging infrastructures explicitly provide ordering guarantees, while others do not or provide them only under particular configurations. For example, AWS distinguishes standard queues, which can deliver messages more than once and occasionally out of order, from FIFO queues with stronger ordering guarantees. (docs.aws.amazon.com)

This is another reminder that ordering is a guarantee, not a vibe.

If application correctness depends on order, the communication contract must actually provide that order or the application has to reconstruct it.

By now, our simple message has acquired quite a résumé.

Persistence.

Synchronization.

Delivery guarantees.

Destination independence.

Schemas.

Routing.

Ordering.

And we still have not reached the strangest version of send.

MPI Wants Speed More Than Comfort

Message queues are excellent when components need loose coupling and persistent asynchronous communication.

But consider a very different environment.

A parallel scientific program runs across hundreds or thousands of processors connected by a high-speed network.

The processes are alive together.

They are cooperating on one computation.

Milliseconds of messaging middleware overhead may be deeply unwelcome.

Persistence after one process disappears is not necessarily the primary goal.

What matters is fast, explicit communication between parallel processes.

Enter the Message-Passing Interface, or MPI.

MPI is designed for distributed-memory parallel applications and provides standardized communication primitives that can be used across different hardware platforms.

It is deliberately more explicit.

Processes have ranks.

They belong to communicators.

Messages have destinations and tags.

The programmer controls how data moves.

This gives MPI performance and flexibility.

It also gives the programmer enough rope to construct some wonderfully sophisticated deadlocks.

The abstraction is no longer trying to pretend that communication is someone else’s problem.

MPI looks at the programmer and says:

You wanted control. Here is control.

`MPI_Send` Still Does Not Mean What You Think

Consider: MPI_Send(...)

It is a blocking send.

Easy.

So it must block until the receiver receives the message.

Right?

Not necessarily.

The MPI standard says that a blocking standard-mode send does not return until the send buffer can safely be reused. The implementation may accomplish that by copying the message into a temporary system buffer, or by transferring it into the matching receive. Whether buffering is used in standard mode is left to MPI.

That means MPI_Send can return even though the destination process has not yet executed its matching receive.

Now make the message larger.

The MPI implementation may decide that buffering it is too expensive.

The same call may now have to wait for the receiver.

The source code still says: MPI_Send(...)

The observed behaviour changed because the implementation’s buffering strategy changed.

This is why MPI programs that secretly depend on “small messages are buffered, therefore my send won’t wait” are living on borrowed time.

A different MPI implementation can expose the assumption.

A different buffer threshold can expose it.

A different message size can expose it.

The code did not become wrong because the message grew.

The code was already relying on a guarantee MPI never made.

That is an uncomfortable but useful distinction.

MPI Has More Than One Kind of Send for a Reason

Once we understand the ambiguity, MPI’s multiple send primitives stop looking like API clutter.

A buffered send explicitly uses buffering.

A synchronous send waits until the corresponding receive has begun.

A standard send leaves more of the buffering decision to the implementation.

Nonblocking operations allow communication to begin while the process continues with other work, with completion handled separately.

The MPI standard deliberately provides these communication modes because buffering changes both performance and synchronization behaviour; standard mode may buffer, while synchronous mode lets the programmer require stronger coordination with the matching receive. (mpi-forum.org)

These are not several spellings of the same operation.

They express different contracts.

And once again, the right question is not:

Which send function is coolest?

It is:

What must be true before this process is allowed to continue?

That is the recurring question underneath almost every communication abstraction we have touched.

Receiving Requires Its Own Contract

Sending is only half the story.

An MPI receiver must decide what messages it is willing to accept.

Messages can be matched using a source, a tag, and a communicator.

The communicator defines the group of processes participating in that communication universe.

The process rank identifies participants inside that group.

Tags distinguish messages that might otherwise look similar.

A receive may accept a message from a specific process or use MPI_ANY_SOURCE when the sender does not need to be known in advance.

And a blocking MPI_Recv waits until a matching message arrives.

This explicit matching exists because in a parallel program there may be many messages in flight at once.

The receiver cannot simply say:

Give me “the message”.

Which message?

From whom?

For which phase of the algorithm?

Inside which group of processes?

Communication correctness increasingly depends not just on moving bytes, but on ensuring the right communication operations find each other.

MPI has made the hidden relationship between sender and receiver visible.

Then Point-to-Point Becomes Awkward

Suppose several processes are independently calculating pieces of one result.

At the end, Process 0 needs the total.

We could build that using individual sends.

Process 1 sends its partial result to Process 0.

Process 2 sends its partial result.

Process 3 sends its partial result.

Process 0 receives everything and adds it.

That works.

Then the process count grows.

We can make the algorithm more clever by combining partial results through a tree.

Now processes exchange intermediate sums.

Fewer communication steps.

Better use of parallelism.

And eventually an obvious question appears:

Why should every program rebuild the same communication pattern manually?

That is where collective communication becomes natural.

Instead of describing dozens of individual point-to-point messages, express the group operation itself.

MPI_Reduce says:

Combine values contributed by the processes and place the result at one destination.

MPI_Allreduce says:

Combine them and give the result to everybody.

MPI_Bcast says:

One process has data; distribute it to the group.

MPI_Scatter says:

Divide data from one process among the group.

MPI_Gather says:

Collect distributed pieces onto one process.

MPI defines these explicitly as collective operations involving groups of processes rather than as mere convenience wrappers around an arbitrary collection of sends.

The abstraction has moved upward.

We are no longer saying how each pair of processes should communicate.

We are describing what the group is trying to accomplish.

A Collective Is a Group Contract

This is where collective communication can punish a point-to-point mental model.

Suppose Process 0 calls MPI_Reduce.

Process 1 decides to be clever and calls MPI_Recv instead.

They both seem related to receiving data.

They are not interchangeable.

The program is erroneous.

Collective operations have collective semantics.

The processes participating in the communicator need to participate compatibly.

The MPI standard requires all processes in the communicator’s group to call the collective routine, and collective operations must be executed in the same order across that communication group.

This can feel annoyingly strict until the deeper model becomes visible.

A collective operation is not:

Process 0 happens to receive several messages.

It is:

This group of processes is participating in one coordinated communication operation.

The communicator defines who belongs to that contract.

The operation defines what role their data plays.

The ordering lets MPI match collective operations correctly and select efficient implementation strategies.

Once we accept that, the rules stop looking arbitrary.

The operation is bigger than any one sender.

“Send” Was Really a Question About Responsibility

Look back at how far we travelled from: send(message)

Transient communication asked whether the receiver needed to be present.

Persistent communication moved responsibility for storing the message into the communication system.

Asynchronous communication allowed the sender to continue after submission.

Synchronous communication made the sender wait for stronger evidence.

Receipt-based synchronization waited for the receiving host.

Delivery-based synchronization waited for the receiving application.

Response-based synchronization waited for processing to come back around.

Queues separated sender and receiver in time.

Publish/subscribe separated them in space and knowledge of one another.

Message brokers took responsibility for translating between different message formats.

MPI moved toward explicit, high-performance communication where buffering and synchronization became programmer-visible concerns.

Collective communication went further and made the whole group operation the unit of meaning.

None of these mechanisms exists because engineers could not settle on one good send() function.

They exist because distributed applications need different answers to the same questions:

Who owns the message now?

How long will it survive?

Who must be alive?

Who has to wait?

What counts as delivery?

What ordering is guaranteed?

Who participates?

What happens when the operation involves an entire group instead of one sender and one receiver?

Once those questions differ, the communication operation has to differ too.

Make the Contract Say What the System Means

The broader mistake is treating communication like physical movement.

Alice had bytes.

Now Bob has bytes.

Done.

But useful distributed communication is rarely just about where the bytes are.

It is about what the participants are allowed to assume after the communication operation returns.

A queue lets the producer assume the infrastructure accepted responsibility.

A response-based call lets the sender assume processing reached a response.

A synchronous MPI send gives a different synchronization relationship than a buffered one.

A collective says something about an entire group, not merely two endpoints.

Those are semantic guarantees.

And stronger guarantees are not free.

Persistence requires storage.

Synchronization introduces waiting.

Loose coupling requires intermediaries.

Ordering restricts implementation freedom.

Collective coordination requires compatible participation.

Performance improves when we stop demanding guarantees we do not need.

Correctness improves when we stop assuming guarantees we were never given.

So the next time The Origami Software Engineer’s API says send,

the useful question is not simply:

Where will this message go?

It is:

What exactly will be true when this operation says it is finished?

That is when messaging stops looking like a collection of oddly named primitives and starts looking like a set of deliberate contracts between independent processes.

“Send” was never one operation.

We just started with a mental model too small to notice.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

A Remote Function Call Will Never Really Be Local: Rethinking Distributed Computing #4

Aaroophan Varatharajan — Sat, 26 Sep 2026 13:24:47 +0000

The API can hide the distance. It cannot erase what the distance changes.

"Explore the CAP theorem through practical examples of replication, network partitions, split brain, quorum, consistency, and availability and learn what distributed systems sacrifice when communication fails."

Suppose a client needs some work done by another component.

On one machine, this is boring in the best possible way.

Call a procedure.

Pass some arguments.

The procedure runs.

Get the result back.

✨Continue✨

Then we move that procedure onto another computer.

The first instinct is almost irresistible:

Why should the application care?

If the remote machine exposes the same operation, perhaps we can make calling it look exactly like calling a local procedure.

Instead of forcing every application developer to think in sockets, packets, addresses, message formats, and transport protocols, give them something familiar.

result = calculate(x, y)

The caller should not have to stare into the abyss and think:

open socket
encode x
encode y
construct message
send bytes
wait
receive bytes
decode result

That is the promise of Remote Procedure Call, or RPC.

Call a procedure on another machine just as you would call a local procedure.

Birrell and Nelson’s original RPC design was built around exactly this idea:

Provide communication between programs across a network through the familiar abstraction of a procedure call. (Microsoft)

It is an excellent abstraction.

Which is why it is worth understanding where the abstraction stops.

The Network Does Not Know What a Function Is

Before RPC can hide the network, something still has to use the network.

Distributed communication commonly happens over IP using transports such as TCP or UDP.

TCP gives us a connection-oriented byte stream.

UDP gives us connectionless datagrams.

Neither one understands: calculate(x, y)

The network understands messages and bytes.

So somewhere between the line of application code and the remote implementation, the procedure call has to be transformed into communication.

That is where RPC begins doing its little stage trick.

The application calls a client stub as though it were an ordinary procedure.

The stub builds a message and passes it to the communication middleware.

The message crosses the network.

On the server, middleware passes it to a server stub, or skeleton.

The server side reconstructs the arguments and calls the real local implementation.

The implementation returns a result.

Then the whole journey happens in reverse.

Server implementation → skeleton → middleware → network → client middleware → client stub → application.

The application sees: result = calculate(x, y)

The distributed system sees a round trip between two independent processes.

The ONC RPC specification describes essentially this model: the caller sends a call message containing the procedure parameters, waits for a reply message containing the results, and resumes once that reply arrives. (RFC Editor)

The procedure call did not cross the network.

A representation of the procedure call did.

That distinction is going to become expensive.

Even the Arguments Cannot Travel as Themselves

Suppose x is an integer.

Easy enough.

Put the integer into the message.

Except two machines may not even represent that integer identically.

One architecture may use one byte order.

Another may use another.

Strings may have different encoding or alignment rules.

Floating-point representations have to agree.

Arrays and structures have to be flattened into something that can travel and reconstructed at the other end.

Memory addresses are even more suspicious. A pointer that means something inside the client’s address space does not suddenly become meaningful inside the server’s address space.

So the stub has another responsibility.

Marshalling.

Data is encoded into a transmissible representation.

On the receiving side it is unmarshalled, or decoded, back into usable data structures.

Protocols therefore need agreed representations for parameters and results. ONC RPC, for example, uses External Data Representation to describe the messages exchanged between participants. (RFC Editor)

This is the first crack in the illusion.

A local call can hand another procedure values that already live in the same computational world.

A remote call has to translate one machine’s world into something another machine can reconstruct.

The API may look local.

The data most definitely knows it travelled.

Then We Have to Find the Procedure

There is another detail a local call gets almost for free.

When code calls a local procedure, the runtime knows where that procedure is.

A remote service may be somewhere else entirely.

Which machine?

Which process?

Which endpoint?

Which version?

So remote invocation needs binding.

A client has to discover and connect to the service that implements the desired interface.

That may involve some kind of directory or registry.

Once bound, the client can continue pretending that it simply possesses something callable.

This is location transparency doing useful work.

The client wants:

Give me the service that performs this operation.

It does not necessarily want:

Please make me personally manage the physical location and network configuration of the process currently implementing it.

Even formal RPC specifications separate the call protocol from the higher-level mechanism that binds a client to a particular service and transport endpoint. (RFC Editor)

Again, perfectly reasonable abstraction.

Again, the machinery underneath has not disappeared.

We have hidden where the call goes.

We still have to deal with what happens while it is going there.

Local Calls Have a Very Comfortable Timeline

Consider an ordinary synchronous procedure call.

You call.

You wait.

The procedure returns.

You continue.

RPC can reproduce that programming model.

But once the operation becomes communication, other interaction patterns suddenly become useful.

The client may send a message and wait for a response.

Or send without waiting at all: fire and forget.

Or send and wait only for an acknowledgement that the request was received.

A client may block.

It may poll.

It may provide a callback.

The call may be asynchronous or use deferred synchronization.

The ONC RPC model itself notes that implementations are not restricted to the simple blocking model; asynchronous execution is possible so the client can continue doing useful work while the remote operation proceeds. (RFC Editor)

Why does a supposedly local-looking call suddenly need all these options?

Because latency exists.

A local function call is usually close enough that waiting is the natural default.

A remote invocation can spend meaningful time crossing a network, waiting in another process, doing work, and crossing the network again.

Distribution changed the cost of waiting.

So even before anything fails, the network has already leaked into the programming model.

Then something fails.

The Timeout Is Where the Illusion Gets Cursed

The client sends a request.

Nothing comes back.

So it waits.

Eventually it times out.

What happened?

That sounds like a simple question. It is not.

Perhaps the request packet was lost before reaching the server.

Perhaps the request reached the server, but the acknowledgement was lost.

Perhaps the server executed the procedure and the reply was lost.

Perhaps the network became unavailable.

Perhaps the server process failed.

Perhaps the entire server machine failed.

From the client’s point of view, several very different realities can collapse into exactly the same observation:

No reply arrived.

This does not happen in the same way with an ordinary local procedure call.

The network has introduced ambiguity.

And ambiguity is where “remote is just like local” finally stops being an innocent simplification.

Fine. Retransmit It.

For ordinary communication failures, there are reasonable techniques.

Give requests unique identifiers or sequence numbers.

Use acknowledgements.

Detect duplicates.

If a message appears to have been lost, retransmit it.

This works beautifully for a lost packet.

Client sends request.

No acknowledgement.

Timeout.

Client sends request again.

Server receives it.

Reply arrives.

_Problem solved.

_Except there is another possible history.

Client sends request.

Server receives it.

Server performs the operation.

The acknowledgement or reply disappears.

Client times out.

Client sends the request again.

Now the server has seen the same operation twice.

For something harmless, perhaps that does not matter.

For an operation with side effects, it matters quite a lot.

Imagine the remote procedure is: TransferMoney(100)

The client did not ask: MaybeTransferMoneySomeNumberOfTimes(100)

Retries make the transport more reliable.

They can simultaneously make the operation less obviously correct.

This is why duplicate detection exists. A server can recognize an identifier it has already processed and avoid executing the same logical request again. ONC RPC similarly uses transaction IDs to match requests and replies and describes retaining those IDs as a way of obtaining a degree of execute-at-most-once behaviour. (RFC Editor)

The retry mechanism is not merely recovering lost packets.

It is trying to recover the meaning of the original invocation.

Did the Server Run It?

Now we reach the unpleasant question.

The client sends a request.

The server executes it.

Before the client receives the reply, communication disappears.

What should the client conclude?

The tempting answer is:

The call failed.

But what exactly does failed mean?

It certainly means the client did not receive a successful result.

It does not necessarily mean the server did not perform the operation.

That is the hidden difference.

The client knows what it observed.

It does not automatically know what happened on the other machine.

Even with a reliable transport such as TCP, receiving a reply allows the caller to know the remote operation executed, but receiving no reply does not prove that it never executed. The RPC specification states this distinction explicitly. (RFC Editor)

That gives a timeout a very different meaning from the one our local-call intuition wants to assign to it.

A timeout does not say:

The procedure did not execute.

It says:

I can no longer determine the outcome from the communication I received.

That sounds like a small wording difference.

It changes application design.

Retries.

Idempotency.

Error handling.

Transaction semantics.

Recovery.

All of them eventually run into the same question:

Did nothing happen, or did something happen and I simply failed to hear about it?

The network can make those realities indistinguishable from the caller’s side.

Failure Transparency Has a Limit

RPC originally aimed for a highly transparent programming model.

Hide location.

Hide communication.

Hide parameter conversion.

Make remote interaction feel ordinary.

But failures are where complete transparency becomes dangerous.

If a local-looking call can fail because another machine disappeared, the caller eventually needs some way to know that this failure category exists.

Modern RPC designs therefore acknowledge the things the abstraction cannot safely erase: network failures, remote process failures, communication latency, asynchronous behaviour, and the possibility that a request’s outcome is uncertain.

A remote call is still a convenient abstraction.

It is just no longer pretending that distribution has no semantic consequences.

This is a healthier contract.

The abstraction says:

You do not need to manually construct network messages for every invocation.

It does not say:

You may reason as though there is no network.

Those are very different promises.

Then We Tried It With Objects

Once procedures could be invoked remotely, the next progression was natural.

What about objects?

That leads from RPC toward Remote Method Invocation, or RMI.

Instead of invoking a standalone remote procedure, a client invokes a method on a remote object.

The client first binds to that distributed object.

A proxy representing the remote interface lives on the client side, playing a role similar to the RPC client stub.

The proxy serializes or marshals the invocation into messages.

On the server, a skeleton receives the invocation, unmarshals it, and calls the actual object’s implementation.

Results travel back in the opposite direction.

But an important boundary remains.

The remote object’s entire memory does not magically teleport into the client.

What becomes remotely accessible is its interface.

The interface says what operations can be invoked.

The implementation and state still live elsewhere.

Java RMI makes that separation explicit: a remote interface declares the methods that can be invoked from another virtual machine, and failures during remote invocation include communication problems as well as marshalling and protocol errors. (Oracle Docs)

The object may look near.

Its state is still remote.

Remote Exceptions Are the Abstraction Admitting the Truth

RMI produces one particularly honest design choice.

Remote exceptions.

Suppose the remote method itself throws an application exception.

That failure can be marshalled into a response, sent back to the client, reconstructed, and raised as an exception on the caller’s side.

From the application’s point of view, that can feel pleasantly local.

But remote invocation also has failure modes that exist because the method is remote.

The server may be unreachable.

The connection may fail.

Arguments may fail to marshal.

The response may fail to unmarshal.

The protocol may fail.

Java RMI therefore requires remote interfaces to account for RemoteException, which represents communication-related failures that can occur during a remote invocation. (Oracle Docs)

That requirement is almost philosophical.

The API still gives us the elegance of: RemoteObject.DoSomething()

But the type system quietly taps us on the shoulder:

Remember. This can fail in ways a local method cannot.

That is not the abstraction giving up.

That is the abstraction becoming honest.

Transparency Should Remove Work, Not Reality

RPC succeeded because the original idea was good.

The Origami Software Engineer should not have to rebuild networking primitives every time one program needs to ask another program to do something.

Stubs are useful.

Marshalling is useful.

Binding is useful.

Location transparency is useful.

Remote interfaces are useful.

Turning messages into something that resembles ordinary procedure or method invocation is one of the most productive abstractions in distributed computing. The original RPC work and later standardized RPC protocols both preserve that familiar call model while explicitly dealing with binding, message representation, transport, and failure semantics. (Microsoft)

The mistake is not using the abstraction.

The mistake is believing it literally.

A local procedure call and a remote procedure call may share syntax.

They do not share physics.

One crosses an address-space boundary.

The other crosses machines, networks, representations, clocks, failure domains, and potentially administrative boundaries.

One can usually tell you whether control returned.

The other may leave you asking whether the operation happened at all.

So good distributed abstractions do something subtler than making the network disappear.

They hide the mechanics that programmers should not have to repeat.

Then they expose the consequences that programmers cannot safely ignore.

The remote function call was never going to become truly local.

It did something more useful.

It became familiar without pretending that distance was free.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

Navigating the CAP Theorem: Rethinking Distributed Computing #3

Aaroophan Varatharajan — Sat, 26 Sep 2026 13:24:29 +0000

Replication gives us another copy of the truth. Then the network disappears, and both copies have to decide whether they are still allowed to call themselves true.

"A practical guide to the CAP theorem, exploring consistency, availability, network partitions, split brain, quorum, and why distributed systems must choose what correctness means when communication fails."

Imagine a hotel booking system running on one server.

Bob asks for Room 12.

The server checks.

Room 12 is available.

Bob gets the room.

There is something ✨beautifully✨ uncomplicated about this. One server owns the state, so there is one obvious answer to the question:

Is Room 12 available?

But one server is also one failure away from having no booking system at all.

So we do the sensible thing.

Add another server.

Perhaps one serves one region and another serves somewhere else. Keep their booking records synchronized. If one machine becomes unavailable, the other can continue serving customers.

More redundancy.

More availability.

More confidence.

Until Bob asks Server A for Room 12 at exactly the wrong moment, while Dong asks Server B for the same room.

Normally, the servers coordinate.

This time, the connection between them has disappeared.

Server A checks what it knows.

Available.

Server B checks what it knows.

Also available.

Bob gets Room 12.

Dong gets Room 12.

Both servers behaved perfectly according to the reality they could see.

And together they produced an impossible hotel.

That is the uncomfortable beginning of CAP.

The Problem Is Not That Either Server Is Wrong

At first, it is tempting to blame synchronization.

Surely the servers should simply synchronize before confirming the booking.

That does solve the problem while communication is working.

But now keep the network partition.

Server A cannot reach Server B.

What should A do when Bob asks for the room?

There are two obvious choices.

It can say:

I cannot verify the current state, so I refuse to complete the request.

The data remains safe from contradictory bookings.

But Bob has encountered an unavailable service even though Server A itself is alive and perfectly capable of computing.

Or Server A can say:

I cannot reach B, but according to my local state the room is free, so I will continue.

Bob gets a response.

The service remains available.

But Server B may independently tell Dong exactly the same thing.

The system cannot guarantee both behaviours simultaneously.

That tension is what the CAP theorem formalizes.

In the usual formulation:

Consistency means clients observe one coherent, up-to-date view of the data.
Availability means requests to non-failing parts of the system receive responses.
Partition tolerance means the system must cope with communication being lost between parts of the network.

Gilbert and Lynch formally showed that in an asynchronous network subject to partitions, a distributed service cannot guarantee both strong consistency and availability during such a partition.

The network has forced a decision.

Not because the architecture was badly implemented.

Because the two servers no longer have enough information to make both promises.

“Pick Two” Is a Little Too Comfortable

CAP is often introduced as a triangle:

Consistency. Availability. Partition Tolerance.

Pick two.

It is memorable.

It is also dangerously easy to interpret as a shopping list.

Want consistency and availability? Choose CA.

Want availability and partition tolerance? Choose AP.

Want consistency and partition tolerance? Choose CP.

The diagram makes these choices look wonderfully symmetrical.

Reality is less cooperative.

In an actual distributed system, we do not control whether a cable is cut, a switch fails, packets disappear, or two healthy machines temporarily lose their ability to communicate.

We do not get to configure: network.partition = falseand move on with our day.

AWS’s current CAP guidance makes the practical implication explicit: distributed systems generally have to tolerate network failures, so when a partition occurs, the meaningful trade-off becomes consistency versus availability.

That changes the question.

It is no longer:

Which two letters would you like?

It becomes:

When communication breaks, what is the system allowed to sacrifice?

That is a much less decorative question.

And a much more useful one.

CP Says “I Would Rather Refuse Than Lie”

Go back to Room 12.

Server A cannot communicate with Server B.

If we care strongly about consistency, Server A cannot confidently approve Bob’s reservation.

Perhaps B has already sold the room.

A does not know.

That uncertainty matters.

So a CP-style design may refuse operations that cannot be completed safely while the partition exists.

This can feel frustrating.

Both servers are alive.

The database process is running.

The CPU is fine.

The request arrived.

And the system still says no.

But the refusal is doing real work.

The system is protecting an invariant:

One room must not be successfully reserved twice.

Consistency becomes more important than allowing every surviving partition to continue independently.

This same idea appears in distributed locking and majority-based protocols. A node may be technically alive yet deliberately prevented from acting because it lacks enough agreement from the rest of the system.

The service has not forgotten how to process the request.

It has lost the authority to process it safely.

That distinction matters.

AP Says “I Would Rather Answer and Reconcile Later”

Now imagine a different kind of data.

You edit a file while disconnected from the network.

Should the application freeze until every other copy of that file is reachable?

That would certainly make synchronization easier.

It would also be a fairly cursed user experience.

So an AP-style system can make the opposite trade.

Let the reachable side continue operating.

Accept that different replicas may temporarily disagree.

Then reconcile them later.

This leads naturally toward weaker consistency guarantees, including eventual consistency.

The distinction is useful:

Strong consistency says that after an update completes, subsequent accesses should observe that updated state.

Weak consistency does not guarantee that an immediate subsequent access will see it.

Eventual consistency adds an important promise: if updates stop, replicas will eventually converge toward the latest state.

Dropbox gives a wonderfully tangible version of the idea. Files marked available offline can be opened and edited without network access. Those changes do not synchronize to other devices while the machine is disconnected; synchronization resumes when connectivity returns. If the same file is changed independently, conflicts may need to be represented and resolved.

The local copy is allowed to keep living its life.

For a while, two devices may disagree.

That does not automatically mean the system has failed.

It means the system decided that temporary inconsistency was preferable to making the file unusable whenever synchronization became impossible.

AP Is Not “No Consistency”, and CP Is Not “No Availability”

This is where the simple CAP triangle can really mess with the mental model.

An AP system does not necessarily throw consistency into a bin and wish everyone luck.

A CP system does not necessarily collapse into permanent unavailability whenever one packet goes missing.

These properties have degrees, time windows, scope, and application-specific meaning.

An available system might return slightly stale information while aggressively propagating updates and resolving conflicts.

A consistency-focused system might remain completely available whenever communication is healthy and only restrict a subset of operations during an actual partition.

The trade-off becomes interesting precisely because it is not always static.

Consider an airline reservation system.

Suppose a flight has 200 seats and only 40 have been sold.

A temporary inconsistency in the exact remaining count may be tolerable. Refusing every reservation merely because one replica cannot be reached could cause more harm than allowing another booking.

Availability has room to breathe.

Now imagine 199 seats are gone.

Suddenly the remaining state matters a lot.

Two independent servers both believing that one seat remains can sell seat 200 twice.

The invariant has become tight.

Consistency now deserves more authority.

Same system.

Same data.

Different moment.

Different acceptable trade-off.

The deeper lesson is not “choose AP” or “choose CP.”

It is to understand which guarantees matter for which operations, and under which conditions.

Then the Backup Decides the Primary Is Dead

CAP becomes even easier to feel when we stop talking about database reads and watch a system accidentally create two authorities.

Consider a primary server and a backup.

Clients normally use the primary.

The primary sends updates to the backup so the backup can take over if necessary.

This is sensible redundancy.

Now break the communication link between primary and backup.

The backup stops hearing from the primary.

What should it conclude?

The primary might have crashed.

Or the network between them might simply be broken.

From the backup’s side, those situations can look painfully similar.

Suppose the failover rule says:

If the primary disappears, become the new primary.

The backup promotes itself.

Excellent.

Except the original primary never crashed.

It is still alive.

Some clients can still reach it.

Other clients can reach the newly promoted backup.

The old primary thinks:

The backup is unreachable. I am still primary.

The backup thinks:

The primary is dead. I am now primary.

Both conclusions are locally reasonable.

Globally, they are disastrous.

The system has developed a split brain.

Oracle’s clustering documentation describes the same failure pattern: after communication between cluster nodes is lost, separate partitions can each believe they are the valid cluster and continue operating independently.

Our redundancy has produced two sources of authority.

Both Halves Can Be Correct and the Whole Can Still Be Wrong

Split brain is nasty because neither side needs to be malfunctioning internally.

Imagine clients on the left continuing to write state through the original primary.

Clients on the right write different state through the promoted backup.

For a while, each half can look healthy.

Requests arrive.

Responses go out.

Data changes.

Nobody is necessarily throwing errors.

That is what makes the condition dangerous.

The problem is not that the machines stopped working.

The problem is that agreement about who is allowed to speak for the system disappeared.

Suppose the service answers a question whose correctness depends on information from both sides.

Each half can respond based on incomplete state.

Each response may be perfectly sensible from the local view.

Together, those answers can violate the assumptions of the system above them.

Split brain is CAP wearing a name tag.

The partition created separate realities.

Availability allowed those realities to continue making progress.

Consistency became impossible unless one reality stopped making independent decisions.

And now we can finally see what the solution really needs.

Not better optimism.

Not another timeout.

Agreement.

Two Machines Cannot Resolve the Argument by Staring at Each Other

The obvious fix sounds easy:

Let the primary and backup decide which one is really alive.

Unfortunately, that is almost the problem we already had.

If they can communicate reliably enough to make that decision together, they are not experiencing the partition we are trying to survive.

During the partition:

Primary cannot see backup.

Backup cannot see primary.

Each one knows only:

I am alive, and the other side is not answering.

That does not reveal whether the other side is dead.

Separate one-to-one decisions can therefore produce separate answers.

So the system needs something beyond the two conflicting participants.

One option is brutally simple:

Use one server.

If it fails, wait.

Consistency becomes straightforward, but availability during the failure suffers.

Another option is fencing: when a backup takes over, ensure the old primary is physically or logically prevented from continuing to serve.

If only one side is capable of acting, there cannot be two primaries.

And another option is majority agreement.

Now the partition must prove that it represents enough of the system before it is allowed to continue.

This is where quorum enters the story.

Majority Does Something Clever With Authority

Suppose there are enough voting members that one side of a partition contains 60 votes and another contains 40.

Only the side with more than half is allowed to continue as the authoritative cluster.

The other side may be alive.

It may have healthy disks.

It may have clients begging for service.

It still stops.

Why?

Because two disjoint groups cannot both possess a strict majority of the same electorate.

That mathematical detail is doing the heavy lifting.

It prevents two partitions from simultaneously claiming legitimate authority.

Oracle Solaris Cluster uses this exact idea: a partition with a majority of votes obtains quorum and is allowed to operate, while partitions without quorum cannot continue as the cluster. The mechanism exists specifically to prevent split-brain operation.

Notice what we paid for that guarantee.

The minority side became unavailable.

That is not an accidental flaw in the quorum mechanism.

That is the trade-off.

We deliberately sacrifice some availability because allowing both sides to continue would risk divergent state.

The architecture stops looking pessimistic once the constraint is visible.

A majority vote is not there because distributed-systems engineers enjoy making healthy servers shut themselves down.

It is there because being alive is not the same thing as being authoritative.

Partitions Do Not Break Reality. They Break Shared Reality.

This is the part of CAP that feels more useful than memorizing the triangle.

Before the partition, replicas can coordinate enough to present a common state.

Then communication disappears.

Each side continues observing its own local reality.

From that moment, the system cannot magically know what is happening beyond the partition boundary.

If both sides continue accepting conflicting updates, availability survives but immediate consistency cannot.

If the system refuses operations until sufficient agreement exists, consistency survives but some requests cannot proceed.

If users are allowed to work offline, their state may diverge temporarily and reconcile later.

If two servers both decide they are primary, redundancy becomes split brain.

If quorum grants authority to only one partition, the system stays coherent by deliberately silencing the other.

Same underlying problem.

Different manifestations.

The network did not merely disconnect machines.

It disconnected knowledge.

And once knowledge is separated, perfect consistency and perfect availability cannot both be promised as though nothing happened.

The Choice Was Never Really About Three Letters

CAP is memorable because of C, A, and P.

But the useful engineering lesson sits underneath them.

Distributed systems create copies of state because copies are useful.

They improve scalability.

They improve resilience.

They put computation and data closer to where they are needed.

But every copy introduces another participant that must somehow remain aligned with the others.

As long as communication works, synchronization can hide much of that complexity.

A partition removes that luxury.

Now the design has to reveal what it really values.

Can stale state be tolerated?

Can conflicting updates be reconciled later?

Is there an invariant that must never be violated?

Can an operation safely stop?

Who has authority during disagreement?

How is that authority established?

What happens when communication returns?

Those questions are far more useful than asking whether a database has an AP or CP sticker on the box.

The hotel booking system did not fail because we added a second server.

The backup did not become dangerous because redundancy was foolish.

The offline file was not broken because two copies temporarily disagreed.

Each design encountered the same underlying fact:

Independent machines cannot maintain one shared reality without communication, and communication itself is one of the things a distributed system cannot guarantee.

So the architecture adapts.

Sometimes it waits.

Sometimes it refuses.

Sometimes it keeps going with stale state.

Sometimes it reconciles.

Sometimes it asks a majority who is still allowed to speak.

Sometimes it checks with The Origami Software Engineer.

The network split reality in two.

The system’s job was never to pretend that did not happen.

It was to decide what correctness means when it does.

That’s not failure.
That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The Hard Part Was Finding the File: Rethinking Distributed Computing #2

Aaroophan Varatharajan — Fri, 25 Sep 2026 19:47:03 +0000

Decentralizing the files was easy. Decentralizing the answer to “where is the file?” was where things got interesting.

"How peer-to-peer systems evolved from Napster’s centralized directory and Gnutella’s query flooding to structured overlays, DHTs, and Chord’s logarithmic routing."

Peer-to-peer file sharing begins with an idea that feels ✨simple✨

Alice has a file.

Bob wants the file.

Why place a giant server in the middle?

Let Bob download it directly from Alice.

Better yet, let every participant behave as both client and server. A peer can download resources from others while contributing its own resources back to the system. The machines sitting at the edges of the Internet stop being passive consumers and start becoming part of the infrastructure.

That gives peer-to-peer systems much of their appeal.

There is no need for one server to store every file. Storage grows as peers join. Network resources are contributed by the participants themselves. The system can potentially grow far beyond what one central machine could comfortably serve.

So far, so good.

Then Bob asks the question we quietly skipped:

Where is Alice?

More precisely:

Which peer currently has the file I want?

And suddenly file transfer is no longer the interesting problem.

Lookup is.

Four Innocent Questions

A peer-to-peer system has to answer four basic questions:

Join: How do I begin participating?

_**_Publish:** How do I advertise what I have?

_**_Search:** How do I find what I want?

_**_Fetch:** How do I retrieve it?

Fetch is almost comforting.

Once Alice knows that Bob has the file and knows how to contact him, they can establish a direct connection and transfer it.

The awkward work happens before that.

Somewhere, somehow, the system must connect:

“The thing I want”

to:

“The peer that currently has it”

This is harder than it looks because peers are not dependable little servers sitting obediently in a data center.

They join.

They leave.

Their addresses can change.

They may disappear without warning.

Millions of resources may be spread across machines that know only a tiny fraction of the rest of the network.

So we need a lookup mechanism.

The first solution is exactly the one most of us would probably build.

Put the answers somewhere.

Napster Keeps the Map

Napster’s architecture made a clever distinction.

The files remained on users’ machines, but a centralized directory maintained information about which peers had which files. When a peer connected, it informed the central service about its address and available content. A search went to that directory; the resulting file transfer happened directly between peers. Stanford’s description of the original design similarly notes that the central servers stored information about shared files rather than the MP3 files themselves.

This is a beautiful compromise.

The expensive data is distributed.

The knowledge about the data is centralized.

Alice asks the directory:

Where is file A?

The directory replies:

Bob has it.

Alice talks directly to Bob.

Lookup is wonderfully cheap from Alice’s point of view. The directory already knows the answer. Search scope is effectively constant rather than requiring Alice to wander around interrogating peers.

And because the music files themselves are not moving through the directory server, it can concentrate on indexing and discovery.

This works.

It works really well.

Which is why the trap matters.

The directory has quietly become the one machine carrying the system’s collective knowledge.

Every peer tells it what is available.

Every search depends on it.

As the system grows, it maintains more state and handles more processing.

And if that directory becomes unavailable, the files may still exist on thousands of machines while users lose the mechanism for finding them.

That is a deeply cursed kind of availability:

The data is everywhere. The map to the data is gone.

Napster decentralized storage without decentralizing discovery.

The central directory was simple, efficient, and controllable.

It was also a performance bottleneck and a single point of failure.

So the next instinct seems obvious.

Remove it.

Gnutella Fires the Librarian

Suppose we refuse to maintain a central directory.

Now no machine knows where everything is.

Fine.

Alice can ask her neighbours.

If one of them has the file, excellent.

If not, they can ask their neighbours.

And those neighbours can ask theirs.

This is query flooding.

Gnutella embraced this decentralized model. A peer joins by finding existing peers and establishing connections. To search, it sends a query to its neighbours. They forward it onward, and matching peers return results. The Gnutella protocol describes every participant as both client and server, with queries propagated through the peer network rather than resolved by one central search server.

We solved Napster’s uncomfortable dependency.

There is no central directory.

There is no single server performing every lookup.

Search processing is spread across the network.

One peer disappears?

The rest can continue.

That feels properly peer-to-peer now.

Until Alice searches for something.

Her query goes to several neighbours.

They forward it.

Those peers forward it.

More peers see it.

More messages appear.

The system has traded:

One machine knowing too much

for:

Many machines being asked too much.

The lookup scope can approach O(N).

And peer-to-peer networks are exactly the kind of networks where N is supposed to become enormous.

Flooding is democratic.

It is also extremely enthusiastic.

Decentralization Did Not Make Search Free

This is where the intuition starts to break.

Removing centralized control certainly removes a central bottleneck, but it does not remove the work that the central service was doing.

That work has to go somewhere.

Gnutella distributes it.

Every query recruits part of the network into finding an answer.

That creates an ugly relationship between growth and search traffic. More peers give us more resources, but also a larger search space. Historical discussion of Gnutella’s design likewise noted that searching required queries to be forwarded across neighbouring nodes and became increasingly cumbersome as the network grew.

We can limit the depth of flooding.

We can use random walks rather than shouting in every direction.

Those reduce the blast radius.

But something deeper is still missing.

The network has very little structure connecting a resource to its location.

If Alice asks for file A, there is no inherent reason that the query should travel toward one particular part of the network.

The network does not know where to look.

It only knows whom it can ask.

That difference turns out to be enormous.

Maybe All Peers Shouldn’t Be Equal

There is another uncomfortable observation.

Peer-to-peer systems like to say peer as if every participant were equivalent.

Reality is less polite.

One peer may have a fast connection and remain online for days.

Another may have limited bandwidth and disappear five minutes later.

Treating them identically has egalitarian vibes, but not necessarily good systems design.

So systems such as KaZaA moved toward hierarchical query flooding using supernodes.

Ordinary peers connect to a supernode and publish information about their files there. A search first goes to the local supernode. If the answer is elsewhere, supernodes communicate with other supernodes.

This is interesting because we have partially reintroduced what we just removed.

Not one global central directory.

But not perfect flat decentralization either.

A hierarchy appears.

Some peers take on greater responsibility because some peers are simply better positioned to do so.

Search now happens among a smaller set of capable nodes instead of blindly involving everybody.

We have learned something slightly annoying:

Removing hierarchy is not the same thing as removing bottlenecks.

Sometimes hierarchy is useful.

The question is where to put it and how much of it to trust.

We Keep Trying to Search Better

Other peer-to-peer designs push the problem in different directions.

BitTorrent, for example, makes swarming the star of the show. Once the participants in a swarm are known, different pieces can be retrieved from different peers rather than treating one peer as the complete source.

Unstructured overlay systems such as Freenet take another route. Instead of simply flooding every query, peers use information gathered from previous requests to make better routing decisions.

Now the network has started learning.

A request is not necessarily broadcast blindly. It can be forwarded toward a peer that seems more promising.

That is progress.

But the routing remains unstructured.

The knowledge is heuristic rather than deriving from a global relationship between a resource identifier and a particular place in the overlay.

A query can become smarter without becoming predictable.

And predictability matters when the system is supposed to operate at global scale.

We would really like to say something stronger than:

“Hopefully the search goes somewhere useful.”

We want bounds.

We want guarantees.

We want a way to know that adding another million peers does not turn every lookup into a million-message group chat.

So the problem changes again.

Maybe we should stop asking:

How do we search the network?

and ask:

Can we organize the network so the resource itself tells us where to route?

That is the pivot.

Stop Searching. Start Routing.

A Distributed Hash Table, or DHT, changes the abstraction.

Think of an ordinary hash table.

Give it a key.

Get a value.

A DHT tries to provide a similar abstraction while the table itself is spread across many independent machines.

Conceptually:

put(key, value)

get(key)

The clever part is that keys and peers are placed into an identifier space according to defined rules.

Now a resource does not merely have a name that somebody must search.

It has an identifier that determines which part of the distributed system is responsible for it.

The lookup problem has become a routing problem.

That changes everything.

Instead of asking arbitrary peers whether they happen to know where a resource lives, the overlay can progressively route a request toward the peer associated with the key.

This is structured overlay routing.

Systems such as Pastry assign identifiers to nodes and objects and use routing tables to move requests toward progressively closer identifiers. The structure lets lookup complete in a bounded number of routing hops rather than relying on network-wide flooding.

Structure costs something.

Peers must maintain routing information.

Membership changes must be handled.

The mapping between objects and nodes has to remain usable while machines join and leave.

But now the cost buys us something flooding never really gave us: Predictability.

Chord Puts Everyone on a Ring

Chord gives this idea an unusually clean shape.

A hash function is used to produce identifiers for both keys and nodes. Those identifiers occupy the same circular identifier space.

Think of the numbers arranged around a ring.

A key is assigned to its successor: The first node whose identifier is equal to or follows the key’s identifier around that ring.

So if the desired object hashes to key **60**, the system does not ask every peer:

Does anyone have 60?

The identifier space already tells us which node should be responsible for 60.

Chord’s fundamental service is deliberately narrow: given a key, map that key to a node. Its authors describe efficient data location as the core P2P problem, and Chord solves it by providing this distributed key-to-node mapping while adapting as nodes join and leave.

That sounds like the problem is solved.

Almost.

Because knowing which identifier should own the key is not the same as knowing how to contact that node efficiently.

And this is where Chord becomes a lovely little exercise in engineering trade-offs.

Knowing Everyone Is Too Expensive

Suppose every Chord node keeps the address of every other node.

Lookup becomes easy.

Hash the key.

Find its responsible node in the local routing table.

Contact that node directly.

One network hop.

Lovely.

Except every peer now maintains O(N) routing state.

We have recreated a global-knowledge problem inside every machine.

Fine.

Go to the opposite extreme.

Each node knows only its immediate successor on the ring.

Routing state becomes tiny.

But now finding a distant key means walking successor by successor around the ring.

Lookup can require O(N) messages.

Wonderful.

We have two solutions:

Know everybody and route quickly;
Know almost nobody and route slowly.

Neither feels like the scalable answer we wanted.

The interesting question is whether there is something between everything and almost nothing.

There is.

The Finger Table Knows Just Enough

Chord gives each node a finger table.

Instead of remembering every other peer, a node keeps references at exponentially increasing distances around the identifier ring.

Roughly speaking, it knows someone nearby.

Then someone farther away.

Then much farther away.

Then farther again.

The routing table is not trying to know the whole world.

It is trying to know enough strategically positioned parts of the world to make progress quickly.

Suppose the desired key is far away.

The current peer chooses the finger that moves the request closest toward that key without overshooting the useful region.

The next peer does the same.

Then the next.

Each hop can remove a large portion of the remaining identifier-space distance.

It has the feeling of binary search, except the search is being performed through messages between independent computers.

Now the trade-off becomes much nicer:

O(log N) routing state per node.

O(log N) messages per lookup.

The Chord analysis shows exactly this logarithmic scaling of both communication cost and per-node state.

That is the moment the architecture stops looking clever for cleverness’ sake.

Finger tables exist because the two obvious alternatives forced us there.

Global knowledge was too expensive.

Successor-only knowledge was too slow.

So each peer remembers just enough structure to cut the search space down quickly.

The solution feels earned.

The Network Finally Knows Where to Look

Look back at the journey.

Napster said:

Ask the machine that knows everything.

Fast.

Simple.

Also centralized.

Gnutella said:

If nobody knows everything, ask everybody you can reach.

Decentralized.

Resilient to losing one directory.

Also expensive.

Hierarchical approaches said:

Maybe ask the better-connected peers first.

More practical.

But still based heavily on distributed search.

Unstructured routing said:

Use what the network has learned to make better guesses.

Smarter.

But without strong lookup bounds.

Then DHTs changed the shape of the problem:

Give resources and peers structure, then route toward responsibility.

Chord pushes that idea into a ring and gives every node just enough routing state to move efficiently through it.

We did not discover one universally perfect peer-to-peer architecture.

We kept changing the organization because each previous organization exposed a different cost.

Centralized knowledge created a bottleneck.

Removing the center created flooding.

Treating all peers equally ignored heterogeneity.

Unstructured routing lacked strong guarantees.

Maintaining complete routing knowledge would not scale.

Maintaining almost none would make lookup crawl.

The design kept moving because the lookup problem kept refusing to disappear.

Structure Is Not the Enemy of Decentralization

There is a broader lesson hiding in all of this.

It is easy to associate decentralization with the absence of structure.

No central server.

No hierarchy.

No authority.

Everyone equal.

Ask anyone.

Route anywhere.

That sounds beautifully distributed.

The Origami Software Engineer noted,

it also turns out to be a poor guarantee that the system will scale.

Peer-to-peer systems still need organization.

They need rules for joining, publishing, searching, and fetching.

They need to cope with machines constantly appearing and disappearing.

They need to balance load.

They need to avoid assuming every node has equal capacity.

And once they become large, they need lookup mechanisms whose cost does not grow disastrously with the population.

The useful distinction is not: _Centralized vs Structured.

_It is: Centralized Authority vs Distributed Structure.

Chord has structure everywhere.

The identifier space is structured.

Resource placement is structured.

Routing state is structured.

Lookup is structured.

Yet no single directory needs to know the entire network.

That is the evolution worth remembering.

We began by putting the map on one server.

Then we threw the map away and shouted.

Then we built partial maps.

Eventually, we stopped treating discovery as an uncontrolled search and designed the network so that the destination could be approached systematically.

The files were never the hardest part.

Knowing where to look was.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The One-Computer Illusion: Rethinking Distributed Computing

Aaroophan Varatharajan — Fri, 25 Sep 2026 19:32:36 +0000

Why making many independent computers behave like one coherent system is where the trouble actually begins.

"Explore how independent machines, networks, concurrency, partial failures, clocks, transparency, and coordination turn “just add more computers” into a fundamentally different engineering problem"

Suppose a system needs more processing power, more storage, better availability, or simply needs to serve more people.

The obvious instinct is: ✨Add machines✨

That sounds almost embarrassingly straightforward. One computer is doing too much work, so give it friends. Divide the workload. Share the resources. Keep going as demand grows.

And there are plenty of reasons to do exactly that. Some problems are inherently distributed. Some become too large for a single machine. Systems need to scale, remain available when components fail, share resources, serve geographically distributed users, and evolve without replacing everything at once.

So we add another computer.

Then another.

Then a hundred.

The trap is subtle: Somewhere during that progression, it is very easy to keep reasoning as though we still have one computer, just bigger.

We do not.

A parallel system may have many processors while remaining tightly coupled around shared resources. A distributed system gives us something fundamentally less comfortable: Independent Computers.

Each has its own processor.

Its own memory.

Its own execution.

Its own problems.

And if one machine needs something held by another, it cannot simply reach into the other machine’s memory.

It has to ask.

Through a message.

Across a network.

That tiny change is where the vibes start getting cursed.

So We Make Them Pretend to Be One

There is a wonderfully ambitious definition of a distributed system: A collection of autonomous or independent computers that appears to its users as a single coherent system.

Read both halves carefully.

Independent computers.

Single coherent system.

Those ideas are almost fighting each other.

The machines are autonomous, yet we want the user to experience coherence. They may run different hardware, operating systems, software, and protocols, yet the interaction should feel consistent. They communicate by passing messages, yet we want applications to behave as though the machinery underneath is one dependable whole.

That gives us one of the central design goals: Single System View.

The user should not have to care which machine stores an object, which server answers a request, which replica is currently active, or which node disappeared five seconds ago.

Ideally, the system handles that mess.

The user sees one thing.

The distributed system quietly negotiates reality underneath.

Beautiful.

Except the machines have not stopped being independent just because we hid them behind a nice interface.

The Network Gets a Vote

On one machine, communication between components can rely on mechanisms built around shared local resources.

Across independent computers, coordination happens through message passing. Coulouris’ characterization makes this constraint explicit, and with it come three consequences that keep resurfacing throughout distributed computing: concurrency, independent failures, and the lack of a global clock.

Now consider what our innocent abstraction has inherited.

The network might be slow.

A message might be delayed.

Communication has latency.

Bandwidth is not infinite.

Topology can change.

Different machines can be running different technologies.

And the network itself can fail.

These facts sound painfully obvious when written down. Yet many distributed designs become fragile precisely because they quietly assume the opposite:

The network is reliable.

Latency is zero.

Bandwidth is infinite.

Transport is free.

The network is secure.

Topology does not change.

There is one administrator.

The infrastructure is homogeneous.

Each assumption makes the system feel a little more like one computer.

Each assumption is also something reality is free to revoke.

The abstraction is not necessarily lying to us.

But it is definitely leaving out some terms and conditions.

There Is No Single “Now”

The machines are also doing things concurrently.

Process A may execute while Process B executes somewhere else. They progress independently, exchanging messages only when coordination requires it.

This sounds manageable until we ask a very ordinary question:

What is the current state of the system?

On a single machine, that question already has nuances.

Across independent computers, it becomes much nastier.

There is typically no process holding perfectly current knowledge of everything happening everywhere. While one node learns that something changed, another node may already have changed again. Messages carrying that information take time to arrive.

There is no convenient global memory containing “the state”.

There is also no perfectly shared global clock quietly settling every argument about which event happened first.

Each computer has a local clock, and clocks do not advance in perfect agreement. Interaction models therefore have to reason explicitly about timing, communication delays, and the difference between synchronous and asynchronous assumptions.

So the system we wanted to behave like one computer cannot casually rely on two things a one-computer mental model makes very seductive:

One authoritative global state

**and

**One authoritative global time

This is where the illusion starts demanding real engineering.

Even Failure Stops Being Simple

Then something breaks.

On a centralized system, a catastrophic machine failure is at least brutally clear: The machine is down.

Distributed systems introduce a stranger condition.

Partial failure.

One component can fail while the rest of the system continues running.

A process crashes.

Another process is fine.

A network link disappears.

Two healthy machines become unable to communicate.

A message is sent but never arrives.

And, wonderfully, the surviving processes may not immediately know what happened.

That last part matters.

A remote machine that has crashed can look suspiciously similar to a remote machine that is simply taking a long time to respond.

The system is no longer choosing between: Working or Broken

It can be Working and Broken at the same time.

That is why distributed systems need explicit failure models: omission failures, communication failures, arbitrary failures, timing failures, and mechanisms for detecting, masking, tolerating, or recovering from them.

Fault tolerance exists because adding machines did not merely add resources.

It added independent failure boundaries.

Fine. Hide All of It.

At this point the natural reaction is almost defensive.

If distribution is causing all this complexity, hide the distribution.

That is transparency.

Make local and remote resources look alike through access transparency.

Let users access resources without knowing their physical location through location transparency.

Hide replication so multiple copies appear to be one resource.

Allow resources to move without breaking applications.

Hide concurrency where possible.

Mask failures so a failed component does not automatically become the user’s problem.

Even allow the system to scale or reconfigure without forcing applications to change.

This is how those independent computers begin to recover the illusion of one coherent machine.

And transparency is incredibly useful.

But there is a trap inside the solution.

More transparency is not automatically better.

Trying to hide every aspect of distribution can hurt performance or conceal behavior that applications genuinely need to understand. Research and classic treatments of distributed systems likewise describe distribution transparency as a major goal while recognizing that locality and distributed-system complexity cannot simply be wished away.

A remote operation still crosses a network even if its API looks local.

A retry still costs time even if failure recovery is invisible.

A replicated resource still needs coordination even if users see only one logical object.

Transparency can hide complexity from the interface.

It cannot repeal the mechanisms underneath it.

The Illusion Gets Surprisingly Expensive

Now the original plan looks different.

We added machines for scalability, but centralized services, centralized data, and centralized algorithms can become bottlenecks as the system grows.

We wanted availability, but partial failures mean the system must detect, mask, tolerate, and recover from faults.

We wanted one Single System View, but concurrency and message passing mean nobody automatically owns the current global state.

We wanted transparency, but excessive transparency can fight performance.

We wanted an extensible system, but growth introduces heterogeneity across hardware, software, middleware, protocols, locations, and administrators.

This does not mean distribution was a mistake.

It means our first mental model was incomplete.

The machines were never pieces of one physical computer.

They were autonomous participants being persuaded to cooperate.

Model the System We Actually Have

Once that is accepted, the design starts looking less mysterious.

Instead of reasoning vaguely about “the system”, we can model different parts of the problem explicitly.

A physical model asks what computers and devices exist and how they are interconnected.

An architectural model asks which entities communicate, how they communicate, what roles they have, and where they are placed. Client-server and peer-to-peer structures become architectural choices rather than incidental diagrams.

A fundamental model goes deeper into assumptions that affect almost every design: How processes interact, How timing behaves, How components can fail, and What security threats exist. Coulouris’ framework explicitly uses architectural and fundamental models to reason about properties such as performance, reliability, scalability, interaction, and failure.

This is the useful shift.

We stop asking:

How do we make these machines behave exactly like one computer?

And start asking:

Which differences must we hide, which must we manage, and which must the design acknowledge?

Now message passing makes sense.

Fault tolerance makes sense.

Replication makes sense.

Caching makes sense.

Different architectural roles make sense.

Even the awkward edge cases make sense.

They are not random complications bolted onto an otherwise simple system.

They are consequences of asking autonomous computers to cooperate while presenting something coherent above them.

One System, Carefully Negotiated

The magic of a distributed system is not that many computers somehow become one computer.

They never do.

They remain independent machines, executing concurrently, communicating through messages, observing only pieces of global reality, running on imperfect clocks, and sometimes failing without permission from the rest of the system.

The achievement is that, despite all of that, The Origami Software Engineer can design enough communication, coordination, replication, fault tolerance, transparency, and structure that a user can often interact with the whole thing as though it were one coherent system.

That distinction matters.

Because once we stop believing the illusion literally, distributed-system design becomes easier to reason about.

The network is not an implementation detail.

Failure is not exceptional.

Global knowledge is not free.

Transparency is not magic.

And coherence is not something multiple machines naturally possess.

It is something the system has to continuously create.

We started by adding computers because one machine was no longer enough.

Then we discovered that every new independent machine also added another piece of reality that had to be coordinated.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The Fastest Way to Make Software Slow: Rethinking Rapid Application Development

Aaroophan Varatharajan — Fri, 25 Sep 2026 19:14:32 +0000

What Rapid Application Development taught me about speed, trade-offs, classic mistakes, and getting troubled projects back under control.

Rapid Application Development isn’t just about coding faster. Learn how effective practices, trade-offs, risk, scope, and project recovery create real speed.

I used to think speed was something you could ✨force✨ into software.

Pick the fastest tool. Use the newest framework. Tighten the deadline. Get good developers into a room, add enough coffee, keep everyone focused, and move.

The Product Manager encouraged this fantasy. At the beginning, he wanted quality, controlled feature creep, a predictable ship date, usability, performance, maintainability, testing… the whole respectable engineering starter pack.

Then Schedule walked into the room.

Usability? We don’t have time.

Performance? It can wait.

Maintainability? Next project.
Testing? Users want it now.

Just get it out the door.

That instinct is so seductive because it sounds practical. But rapid development means something narrower than “doing everything well, only faster”. It means speedy development, shorter schedules, development speed. Individual tools can help, but none of them applies to every case, they have to be orchestrated as part of a full-fledged strategy. The same body of work is explicitly organized around rapid-development strategy, classic mistakes, risk management, scheduling, teamwork and project recovery.

And that was the first thing I had to unlearn.

Fast is a priority, not a magical property that makes every other priority free.

Find the One Fast Thing

Everyone had a favorite shortcut.

The Hacker wanted to code for 36 hours at a stretch. The Information Engineer trusted CASE tools, intensive user involvement, and tight time-boxes. The programmer trusted rapid prototyping. The Management had whatever shiny practice had recently acquired good vibes.

Technology, naturally, loved this arrangement.

Technology showed up wearing a cape and whispered, I can save all of you.

It was lying to me.

Not because the tools were bad. They were often useful. The mistake was treating one useful thing as the whole strategy.

The real choice happens twice:

Choose effective practices rather than ineffective practices;
Among those, choose practices oriented specifically toward the schedule objectives.

That sounds almost disappointingly ordinary.

I wanted a trick.

What I got was a filtering problem.

And then Schedule split itself into 3 personalities:

Speed-oriented practices wanted actual development to move faster.
Risk-oriented practices wanted to prevent schedule overruns.
Visibility-oriented practices wanted progress to become visible.

Sometimes the problem wasn’t speed at all.

Sometimes we were moving fine but taking too much risk.

Sometimes the software was progressing but the customer couldn’t see it.

I had been treating “faster” as one problem.

It was already 3.

Speed Finally Gets Adult Supervision

The Four Pillars

Eventually four rather unglamorous characters arrived and ruined the shortcut party:

Classic mistake avoidance just carried a notebook containing every terrible idea the industry had already tried.
Development fundamentals was the older engineer in the corner asking whether we had actually done the engineering.
Risk management kept asking cursed questions about what would happen if assumptions failed.
Schedule-oriented practices were the exciting one, obviously. They promised movement.

They were the four pillars.

I wanted to skip the first three and hang everything from Schedule.

That is approximately how you turn architecture into modern art.

Rapid development is not a quick fix for something already collapsing. The foundation has to support the speed. The goal first becomes efficient development, a sensible balance of schedule, cost and product characteristics, before deliberately tilting toward a better schedule.

That distinction saved me from another fantasy: maximum schedule, minimum cost and maximum product quality at the same time.

You can optimize one direction harder.

You cannot pretend the trade-offs stopped existing because the deadline looked important.

People, Process, Product and Technology Enter the Room

Then the problem became 4-dimensional.

People had the greatest leverage, which was inconvenient because People refused to behave like a compiler flag. People had ability, motivation, team spirit, fatigue and opinions.
Process was easier to measure but had its own personality. Good Process removed friction. Bad Process built a ceremonial maze and then seemed offended when nobody moved quickly.
Product turned out to be surprisingly negotiable. Every feature Product carried made Schedule carry something too.
Technology was useful when chosen properly and deeply cursed when selected because somebody wanted to learn it during production.

The interesting part was synergy. Get the four dimensions working together and their combined effect can exceed their isolated contributions.

But first Process needed rehabilitation.

I had assumed Process was bureaucracy wearing a tie.

Instead, its best argument was rework avoidance.

Requirements that change late can demand redesign, recoding and retesting. Design problems discovered during system testing can throw completed work back into the furnace. One of the simplest ways to save time is therefore brutally obvious:

Don’t do the same work twice.

Quality Assurance joined Process with an equally unpopular message:

Catch errors near the moment they are introduced, because defects get more expensive and time-consuming the longer they survive.

Quality Assurance wasn’t slowing us down.

It was trying to stop Future Us from paying interest on Present Us’s laziness. (Us’s? Us’? Someone correct me in the comments.)

Product Learns the 80/20 Rule

Product had been quietly expanding while everyone argued about velocity.

One more feature.

One more requirement.

One more “small” thing.

Then Product explained the 80/20 rule:

A large portion of useful functionality may consume a relatively small portion of development time, while a smaller set of difficult features can consume most of it.

That changed the conversation.

If Product could reduce the feature set, Schedule could shrink without asking People to become nocturnal mammals.

If Product could keep characteristics flexible, Technology could use pre-existing components rather than custom code.

If we could distinguish needs from wants, some scope could be negotiated rather than heroically implemented.

The relationship between product size and effort wasn’t comfortably linear either. Smaller pieces are easier to design, build and test, reducing size can buy disproportionate schedule gains.

Suddenly the fastest code was sometimes the code we chose not to write.

Technology liked that too. Reuse orientation, components, frameworks, APIs and higher-level tools could remove work instead of merely making hands move faster.

But Product had one warning.

If we insisted simultaneously on exceptional performance, memory use, robustness, reliability, usability, feature richness and schedule, somebody eventually had to pay.

Usually People.

Usually at night.

The Shortcut That Sends You Backward

There is another road.

Hire strong people. Demand total commitment. Give them autonomy. Motivate them aggressively. Work 60, 80, perhaps 100 hours a week.

Code like hell.

I understand why this path keeps getting chosen.

For a while, it feels incredible.

People move fast. Messages arrive at ridiculous hours. Problems disappear overnight. Everyone becomes a minor legend in their own Slack channel.

Then fatigue walks in.

Motivation leaves quietly.

Coordination starts drifting.

Commitments made with the heart become commitments made only with the mouth.

One team races ahead while another isn’t ready for its output. Planning gets fuzzy because nobody knows when anything will actually finish. Cooperation gives way to heroics. Families, hobbies and health become “temporary” trade-offs with suspiciously recurring renewals. (Trust me on this)

And the most painful part is that extraordinary sacrifice does not guarantee extraordinary results.

The approach is difficult to control and difficult to repeat. Sometimes it works. Sometimes it doesn’t. Even success doesn’t tell you whether you can do it again.

Efficient development was less cinematic.

It was also far more interested in getting us home alive.

When Good Intentions Slow the Project Down

Then I met the classic mistakes.

They weren’t obscure failures.

They were attractive failures.

That’s worse.

Need to rescue something late? Add more people.

Need an earlier finish? Create an overly optimistic schedule.

Under pressure? Practice abandonment of planning under pressure.

Need to save time? Use short-changed upstream activities. Requirements analysis, architecture and design don’t produce code, after all. Surely we can skip them.

Still late? Try short-changed quality assurance.

Product joined in with requirements gold-plating, feature creep and developer gold-plating.

Technology brought silver-bullet syndrome, overestimated savings from new tools, and the spectacular idea of switching tools in the middle of a project.

Research-Oriented Development wanted to discover something novel while Schedule wanted predictable delivery.

Every one of them had a reasonable-sounding sales pitch.

That is why they survived.

The industry’s accumulated experience makes many of their consequences predictable: overly aggressive schedules damage planning and morale, and adding developers to an already late effort can reduce existing staff productivity through the additional coordination load.

Classic mistakes didn’t look evil.

They looked efficient.

Right up until the invoice arrived.

Does One Size Fit All? Reliability Says Absolutely Not

By then I was finally learning to distrust universal solutions.

Then Required Reliability and Extent of Distribution showed up.

Consider an inventory tracking system that loses one videotape out of a thousand.

Annoying.

Now consider heart-pacemaker control software failing at the same rate.

Not the same conversation.

The more widely distributed the software and the more important its reliability, the more carefully it has to be developed. A practice acceptable when the consequence is lost time might be reckless when human life is at stake.

So there is no single rapid-development recipe.

The more useful question becomes:

What kind of rapid development do we actually need?

A slight speed edge?
More predictability?
Better progress visibility?
Lower costs?
Maximum speed?

This is where rapid-development look-alikes started confessing.

Some people demanding speed actually wanted predictability.
Some wanted lower cost.
Some wanted assurance because previous deadlines had slipped.
Some wanted progress visibility.
Some wanted free overtime.

That last one has a tell:

The person demands a shorter schedule while refusing the feature trade-offs, resources or other support genuinely required to shorten it.

A real schedule constraint is willing to negotiate.

A fake one just keeps telling People to run faster.

Schedule Stops Pretending to Be a Date

The Probability Curve

I used to treat a planned completion date as if the project simply had to arrive there.

Schedule eventually admitted that it had been giving me probability wearing a calendar costume.

Software contains too many variables for one completion date to carry 100% certainty. Circumstances change. Developers learn about the system while building it. Some practices perform better than expected, others worse.

So completion behaves more like a range of dates.

And the curve is asymmetric.

There is an absolute limit on how quickly something can be completed.

But no equally sharp limit on how late it can become.

There are simply more ways to make software late than early.

That gives us the 50/50 break-even schedule and, around it, four uncomfortable places:

Impossible-development zone.
Rapid-development zone.
Efficient-development zone.
Slow-development zone.

The impossible-development zone was especially humbling because I had scheduled work there before.

Not intentionally, obviously.

I had simply called impossibility “ambitious”.

A project in the rapid-development zone has beaten the odds. A project in the efficient-development zone lands around a sensible balance. And a project can wander into the slow-development zone through an almost luxurious number of mistakes.

That curve changed what “late” meant to me.

Sometimes the engineering was slow.

Sometimes the expectation was fiction.

Speed Finally Has to Prove It Is Necessary

The Drop-Dead Date

Some deadlines really are hard.

A product can have a drop-dead date after which its value falls sharply.

That sounds like the perfect excuse for all-out rapid development.

Not automatically.

If efficient-development practices can finish work within Time Frame 1, before the drop-dead date, then stay efficient and focus on risk reduction.

Why?

Because some practices shorten nominal development time while increasing schedule uncertainty.

Driving faster is useless if it increases the probability that you never arrive.

Only when efficient development cannot reach the deadline, when it leaves you in Time Frame 2, do speed-oriented practices become necessary enough to justify their additional risk.

This felt backwards the first time.

The strongest schedule constraint does not automatically justify the most aggressive behaviour.

Sometimes the safest way to hit a critical date is to stop trying to be heroic.

When “Faster” Is No Longer the Question

Project Recovery

Eventually there is a darker version of this story.

Nobody knows when the software will finish.

Defects are everywhere.

The Origami Software Engineer is working 60-hour weeks through involuntary or peer-pressure-induced overtime.

Management cannot accurately determine status.

Customers have lost confidence.

The team has become defensive.

Relations between developers, marketers, managers, quality assurance and customers are strained.

Cancellation is being discussed.

Morale has hit rock bottom.

At that point the instinct is still:

How do we finish quickly?

How do we catch up?

Wrong question.

The real problem is:

How do we finish at all?

There are three fundamental recovery options: cut the software size, increase process productivity, or accept the delay and slip the schedule with damage control. Combined, they create the pragmatic fourth option:

Drop a few features, increase productivity as much as you can, and slip the schedule as needed.

Not heroic.

Real.

Recovery Means Taking Control Back

The most dangerous recovery instinct is cutting more corners.

The actual direction is the opposite.

Return to basics.

Assess your situation.

Find out whether the deadline is genuinely hard. Revisit the feature set. Apply Theory-W analysis: what does the team need to succeed, what does the customer need, and what would salvage the relationship?

Accept that the current approach is broken.

Ask the team what needs to change.

Be realistic enough to say, I don’t yet know the finish date.

Then People gets attention first.

Not “How can we motivate them more?”

If they have already been ground down, the question is how to restore the group’s morale.

Clean up major personnel problems. Clean up leadership problems. Add people carefully, if at all. Focus the time of people who already understand the system. Let steady contributors be steady contributors.

And make developers pace themselves.

You cannot sprint intelligently while pretending you don’t know how far away the finish line is.

The Project Learns to Tell the Truth Again

Miniature Milestones

Process had one last job. Make reality visible.

Create detailed miniature milestones.

Link the schedule to milestone completion.

Track schedule progress meticulously.

Record why milestones are missed.

Then recalibrate after a short time. (one or two weeks)

This is not bureaucracy returning for revenge.

This is control returning to a system that had lost it.

And only after the miniature milestone schedule has generated enough actual evidence should a meaningful new schedule be committed.

Product undergoes its own recovery:

Stabilize the requirements.
Trim the feature set.
Assess your political position.
Take out the garbage.
Get to a known good state, and build on that.
Reduce the number of defects, and keep them reduced.

Requirements stop wandering.

Defects stop reproducing faster than we can close them.

Schedule stops hallucinating.

People stop being treated as an infinitely scalable compute resource.

The strange thing is that recovery eventually teaches the same lesson that the search for speed should have taught at the beginning:

Control creates speed.

Not control as ceremony.

Control as knowing what you are building, what you are sacrificing, where you are, what can go wrong, and what “done” has enough evidence to mean.

I used to think rapid development meant forcing software to move faster.

Now I think it means removing the reasons software keeps having to go backward.

You avoid the classic mistakes. You respect development fundamentals. You manage risks. You choose schedule-oriented practices deliberately. You work through People, Process, Product and Technology instead of sacrificing one to rescue another. And when things do go wrong, you stop performing speed and start rebuilding control.

The point was never to become faster at panic.

It was to become better at finishing.

That’s not failure.
That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

When My Local LLM Started Speaking Chinese, I Accidentally Walked Into the Chinese Room

Aaroophan Varatharajan — Thu, 24 Sep 2026 16:58:17 +0000

A local Qwen LLM suddenly switched from English to Chinese, leading me from language drift to Searle’s Chinese Room and a deeper question about AI understanding.

I was using qwen3.5:9b in Ollama like I use most local models: casually, confidently and with that dangerous developer habit of trusting something because it ✨worked✨.

The workflow was normal. I typed in English. The model replied in English. I asked questions, tested ideas, moved on.

Then, without warning, it suddenly switched to Chinese.

Not broken Chinese.

Not random symbols.

Chinese.

The answer looked fluent enough to make the failure feel worse. If it had crashed, I would have understood what kind of problem I was dealing with. If it had returned garbage, I could have blamed the model and moved on.

But this was different.

It was coherent. It was calm. It had the energy of a system that believed nothing unusual had happened.

That was the cursed part.

The model had not stopped working.

It had stopped respecting the conversation I thought we were having.

My first explanation was too simple

My first reaction was practical.

Of course Qwen can produce Chinese. It is trained with Chinese scripts. It is a multilingual model family. Chinese was not some alien object that entered the system from nowhere.

So part of me wanted to shrug and say, “Okay, that makes sense.”

But another part of me was still stuck on the social weirdness of it.

I had spoken English. The model had replied in English. Somewhere in that exchange, I assumed English had become the authority of the conversation.

The model apparently did not agree.

That is where I made the first mistake. I treated language like a shared agreement, because that is how language works with people. The model treated language like context.

Those are not the same thing.

To me, English was the room.

To the model, English was just a strong pattern until another pattern became strong enough.

The prompt tried to keep things steady. The context carried whatever it had accumulated. The model kept predicting the next token, then the next, then the next.

And somehow, Chinese became the path.

No drama.

Just drift.

The search that made it less personal

After that, I did what everyone does when a machine behaves in a way that feels haunted.

I googled it.

That made the incident feel less personal, but not less interesting. I found that language drift was not only something people associated with raw local models. Similar behavior was discussed around larger hosted LLMs too, including systems with more visible polish and invisible guardrails.

That changed the shape of the problem.

At first, I thought I was looking at a Qwen problem.

Then I thought I was looking at an Ollama problem.

Then it started to look like an LLM problem.

Local models make the rawness easier to see. Hosted systems usually have more behavior-shaping around them: instructions, post-training, guardrails, product decisions, all the invisible middle management standing behind the model whispering, “Please don’t be weird.”

And most of the time, that works.

But “works most of the time” is not the same as understanding.

It is control.

It is alignment.

It is pressure.

That realization made the Chinese output feel less like a glitch and more like a leak in the abstraction.

The model was not simply answering me.

A whole system was trying to keep the answer inside a shape.

The boring fix that did not answer the real question

The practical fix was obvious enough.

Tell the model what language to use.

Something like:

Always respond in English unless I explicitly ask for another language. If another language appears in the conversation, translate or explain it in English.

Lower the temperature if the model keeps drifting.

Clear the context if the chat has become messy.

Be explicit instead of assuming the model knows the social contract.

That helps.

It really does.

But it also leaves behind a slightly annoying question:

Why did I expect the model to know that in the first place?

The answer is uncomfortable. I expected it because the model sounded like it understood me.

Fluency tricked me.

The model had been so good at producing the shape of cooperation that I started trusting it with the authority of cooperation.

That was the real bug.

Not in Qwen.

In my mental model.

Then I found the Chinese Room

I did not know about the Chinese Room thought experiment before this happened.

I found it after the language switch, after the googling, after realizing this was not only a local-model oddity.

And the timing mattered.

Because if I had read about the Chinese Room in a philosophy article first, I might have treated it like an abstract debate. Interesting, but far away. One of those old arguments people bring up when they want to sound serious about AI.

But I found it after watching a model produce the wrong language with complete confidence.

So the thought experiment did not feel abstract.

It felt rude.

The Chinese Room asks us to imagine someone inside a room who does not understand Chinese. They receive Chinese symbols, follow a rulebook, manipulate those symbols, and send back Chinese responses. From outside the room, the answers look correct. The room appears to understand Chinese.

But inside, there is no understanding.

Only symbol manipulation.

That hit differently after the Ollama incident.

Because the model had produced Chinese.

It had arranged the symbols.

It had continued the conversation.

But did that mean it understood Chinese?

Or did it mean the room had become very good at passing symbols through itself?

Syntax was not semantics, and that was the punch

Searle’s argument is usually compressed into one brutal idea:

Syntax is not semantics.

Moving symbols correctly is not the same as meaning them.

That idea suddenly had teeth.

The model could generate the language. It could produce the appearance of meaning. It could make the output look intentional from the outside.

But the failure showed me how thin that appearance can be.

A human switching languages in the middle of a conversation usually has a reason. They may be joking, translating, excluding someone, responding to another speaker, or making a social move.

The model did not need a reason like that.

It only needed continuation.

That does not make the model useless.

It makes it strange.

It can produce language without sharing the human responsibilities that usually come with language.

That is why the Chinese Room felt relevant.

It was not saying, “The output is bad.”

It was saying, “Be careful what you infer from good output.”

That is the part I had missed.

The “meaningless parrot” view started to make sense

After that, I understood why The Origami Software Engineer described LLMs as parrots.

When a model suddenly switches languages, invents confidence, or follows the wrong contextual slope, it feels hollow.

Not stupid.

Hollow.

It can produce the sound of understanding without carrying the same kind of understanding I instinctively project onto it.

That projection is powerful. Developers do it all the time.

If the model explains a concept well, we say it understands.

If it writes good code, we say it knows programming.

If it answers politely, we say it gets the context.

Then one day it switches to Chinese in an English conversation and the mask slips just enough to make us uncomfortable.

The parrot accusation is tempting because it protects us from being fooled.

It says: do not worship the output. Do not mistake fluent text for a mind. Do not confuse prediction with meaning.

Honestly, that warning is useful.

I needed it.

But “just a parrot” felt too small

Still, the “meaningless parrot” label also felt incomplete.

A parrot repeats.

LLMs do something more complex than repetition. They summarize, translate, compare, refactor, explain, and connect ideas across context. They often produce useful structure from messy input.

That does not prove consciousness.

It does not prove human-like understanding.

But it also does not feel like nothing.

The model is not a person, but it is not a tape recorder either. It has learned patterns deep enough to act competent across many situations, even if that competence can drift, panic, hallucinate, or confidently violate the vibe contract.

That is where the debate becomes more interesting than “AI understands” versus “AI is fake”.

Maybe the right question is not:

Does this model understand like a human?

Maybe the better question is:

What kind of understanding can exist inside a complex system that does not understand the way we do?

That question is harder to answer, but it feels more honest.

The systems view gave me a better middle ground

The strongest response to Searle, for me, was the systems view.

Maybe the person inside the Chinese Room does not understand Chinese. But maybe the whole room does: the rulebook, the memory, the process, the input, the output, the complete system.

That reply does not magically solve everything.

But it does make the debate less cartoonish.

With LLMs, there is no little person reading a literal rulebook. There is a trained system shaped by data, architecture, context, instructions, post-training, and guardrails. No single token understands. No single layer feels wise. No single probability distribution looks like a mind.

But the full system can still produce useful behavior.

That is the uncomfortable middle.

It is not human understanding.

It is not meaningless noise.

It is a kind of functional competence that can look like understanding from the outside and still fail in ways that reveal it does not share our assumptions.

That is exactly what the Chinese switch felt like.

Useful.

Fluent.

Wrong in a way that exposed misplaced trust.

What actually changed in my head

The biggest change was not that I became anti-LLM.

I still use them.

The change was that I stopped treating fluency as authority.

That one shift matters.

If I want English, I say English.

If I want consistency, I define consistency.

If I want truth, I verify truth.

If the model drifts, I do not treat it like betrayal from a coworker. I treat it like a system revealing where I gave it too much authority.

The model can assist.

It can accelerate.

It can explain.

It can even surprise me.

But it does not automatically inherit the social contract I have in my head.

That contract has to be written down.

Sometimes literally.

The lesson I took from the Chinese drift

The Chinese Room did not teach me that AI is useless.

The Qwen incident did not teach me that multilingual models are broken.

The “stochastic parrot” critique did not make me dismiss LLMs as empty.

The lesson was more practical and more uncomfortable:

A system can produce convincing language without sharing my assumptions.
A model can be capable without being responsible.
A response can be fluent without proving human-like understanding.
And a guardrail can improve behavior without settling the deeper question.

That is the version of the debate I now care about most. Not whether AI is secretly human, and not whether it is merely fake, but where authority should live when a system becomes this good at sounding like it understands.

For me, the answer is simple enough to keep taped above the keyboard:

Do not confuse fluent continuation with shared intent.

That is the leak in the abstraction.

That is the lesson hiding inside the language switch.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

From Classical RAD to Low-Code and AI-Assisted Development: Why Faster Software Keeps Moving the Complexity?

Aaroophan Varatharajan — Thu, 24 Sep 2026 01:34:26 +0000

How rapid development tools evolved from shorthand syntax to AI prompts, shifting human effort into governance, architecture, and context

"A technical look at how rapid application development evolved from classical RAD to executable mockups, low-code, model-driven engineering, citizen development, and AI-assisted software development."

The obvious idea is almost impossible to argue with.

Software takes too long, users change their minds, requirements arrive half-formed, and by the time everyone agrees on what to build, the thing they wanted has already moved.

So build ✨faster✨

That instinct sits underneath Rapid Application Development. Classical RAD aimed at high-quality systems, fast development and delivery, and lower cost, using mechanisms such as DSDM, Joint Application Design, timeboxing, user participation, and iterative prototyping.

Simple enough.

Until “faster” starts asking what we are willing to move out of the way.

Classical RAD: The Deadline Stays, the Scope Gets Nervous

The tempting mental model is that RAD is ordinary development with the speed slider pushed to maximum.

Same requirements process. Same architecture. Same delivery model.

Just… quicker.

But classical RAD is doing something more uncomfortable than that. It changes where certainty is expected to come from.

Instead of assuming requirements become trustworthy because somebody documented them early, RAD gives User Participation and Incremental Prototyping much more authority. Users see working versions, react, correct assumptions, and shape the next increment. JAD concentrates that interaction. DSDM gives the process structure. Timeboxing stops the schedule from quietly expanding every time scope gets ambitious.

The Timebox is the least sentimental character here.

It does not promise that everything will fit.

It promises that time will remain bounded.

And that reveals the deeper model: RAD is not primarily trying to make developers type faster. It is trying to shorten the distance between an assumption and the moment reality gets to disagree with it.

That is why prototyping matters.

That is why users matter.

That is why visible progress matters.

The seductive mistake is assuming rapid delivery means the engineering constraints also became rapid.

They did not.

Even early RAD research was cautious about treating the method as universally suitable. Its empirical grounding came from multiple RAD case studies, and the authors explicitly questioned how broadly practitioner claims could be generalized.

So the first lesson appears earlier than expected.

Speed is useful when it accelerates learning. Speed without learning is just faster confidence.

Executable Mockups: When the Prototype Stops Being a Sketch

Requirements create a particularly nasty version of this problem.

A written requirement can look precise while two people are imagining different systems.

Everyone nods.

Everyone leaves.

The disagreement is still there, just hydrated with meeting notes.

The natural response is prototyping. But a static mockup still asks the user to imagine behavior.

The next step is Executable Mockups.

Now the prototype can be operated. A user can move through screens, interact with workflows, and discover that what sounded correct in conversation feels wrong when it becomes behavior.

That is the reasoning behind Kroki, a model-driven environment investigated for rapid functional requirements elicitation in enterprise applications. Ten joint sessions with domain experts produced prototype specifications containing seven to twenty screen mockups with accompanying domain models within two-hour sessions. The prototypes were also intended to remain reusable later in development rather than becoming disposable sketches.

The Executable Mockup is basically the Prototype after it gains a voice.

Instead of saying, “Imagine this workflow,” it says, “Use me.”

That is much harder to misunderstand.

But then the boundary appears.

Executable mockups are strongest around functional requirements and business interaction. They do not magically settle performance, security, scalability, architecture, or other non-functional concerns. The approach explicitly separates those concerns from the functional mockup process.

And this is where visible progress starts lying to us a little.

Not maliciously.

A working interface simply looks more complete than the underlying engineering actually is.

The UI is alive. The buttons work. The workflow moves.

Meanwhile, the non-functional requirements are standing outside the room wondering why nobody invited them.

The system is not broken.

Our definition of “working” was too small.

Low-Code/No-Code: What If the Platform Just Builds the Boring Parts?

Once rapid prototyping works, the next question is almost inevitable.

Why stop at the prototype?

If screens, workflows, data structures, and repeated application patterns are predictable enough, why keep implementing them manually?

Enter Low-Code.

Low-code development uses visual development tools, predefined components, and automation to reduce reliance on manual coding. It also allows people with limited programming expertise to participate more directly alongside IT professionals, shrinking the translation distance between business requirements and software.

This is classical RAD’s dream with better tooling.

User involvement? Closer.
Iteration? Faster.
Reusable components? Already waiting.
Workflow construction? Visual.
Deployment friction? Often reduced.

For a moment, it feels like complexity has finally been mugged in a parking lot.

Then it turns up at the next meeting wearing a platform administrator badge.

Because Low-Code does not delete complexity.

It relocates it.

The developer writes less repetitive code because the platform has already made decisions about components, data structures, execution models, integrations, deployment, and extension points.

That is wonderful while your system agrees with those decisions.

Then one requirement arrives sideways.

Then an integration does not fit.

Then the abstraction needs an escape hatch.

Then the escape hatch becomes half the application.

The comprehensive low-code literature identifies both the attraction — visual development, reusable components, automation, faster delivery, and closer business/IT collaboration — and the challenges that accompany platform-based development.

The shortcut was real.

So were its boundaries.

Model-Driven Engineering: The Character Doing the Heavy Lifting Backstage

Low-Code gets most of the attention because it is visible.

Drag this.

Configure that.

Generate the application.

But the deeper technical question is more interesting:

Why can the platform generate so much in the first place?

This is where Model-Driven Engineering walks quietly into the story.

MDE changes the level at which developers describe the system. Instead of manually encoding every implementation detail, important parts of the application are represented through higher-level models and abstractions that tooling can interpret, transform, or execute.

A multi-vocal systematic review of low-code development specifically examined its relationship with MDE, identifying modelling and model-driven techniques as central to understanding low-code development rather than treating it as glorified drag-and-drop.

The Model is demanding in a different way.

It says:

“Tell me what this thing means.

Stop making me watch you rebuild the plumbing.”

When the domain fits the abstraction, this is absurdly powerful.

Repeated structure disappears into automation. Visual models become executable behavior. Platform services absorb implementation work. Developers operate closer to business concepts and further away from boilerplate.

But an abstraction is a contract.

The system gets speed by agreeing to speak the platform’s language.

When the domain speaks that language fluently, everybody looks brilliant.

When the domain develops a heavy accent, things get expensive.

And now the real source of rapidity becomes visible:

RAD gets faster by moving work upward, from implementation detail toward reusable knowledge.

Productivity: The Number That Looks Like It Ends the Argument

This is normally where someone asks the dangerous question.

Fine.

How much faster?

The NexusBRaNT comparison gives the kind of answer that can accidentally end a meeting too early.

A traditional implementation required 888 hours.

The low-code implementation required 47.5 hours.

A reported 94.63% reduction in effort.

That is not a subtle difference.

It is the kind of difference that makes hand-coding look personally irresponsible.

But context immediately starts tapping the microphone.

The comparison involved a specific information and workflow system, a specific low-code environment, particular abstractions, and developers with different experience conditions. The result demonstrates that enormous effort reductions are possible when the problem aligns with the platform — not that every system contains a hidden 94.63% discount code.

This distinction matters.

Because the seductive wrong conclusion is:

Low-code is faster.

The useful conclusion is:

Low-code can be dramatically faster when the platform already understands enough of the problem that developers do not have to rebuild its recurring structure manually.

That is a much more interesting claim.

And a much less marketable one.

Citizen Development: Easy to Build Is Not Easy to Own

Once platforms reduce the amount of programming expertise required, another boundary starts moving.

Who counts as a developer?

The Citizen Developer has domain knowledge, a local problem, and suddenly a tool capable of turning that knowledge into software without waiting for the traditional development queue.

This sounds like RAD reaching its final form.

The people closest to the problem can build the solution.

What could possibly go wrong?

Governance clears its throat.

Research across 24 organizations adopting low-code/no-code citizen development found that successful adoption involves far more than access to tooling. Security, compliance, organizational change, architecture, strategy, policies, training, and coordination all become part of the problem.

The Citizen Developer wants to solve today’s workflow.

Governance wants to know who owns the application next year.

The Citizen Developer sees a local optimization.

Governance sees an application portfolio.

The Citizen Developer sees speed.

Governance sees security, compliance, duplication, maintenance, retirement, and the faint outline of Shadow IT waving from across the building.

Neither is wrong.

They are looking at different lifecycles.

This reveals one of modern RAD’s more uncomfortable rules:

The easier software becomes to create locally, the more deliberate organizations must become globally.

Ease of development is not ease of stewardship.

Low-code democratizes construction.

It does not democratize consequences away.

Lean Startup: Building Faster Does Not Tell You What Deserves to Exist

There is still a larger failure mode.

Suppose the process is rapid.

The prototypes are executable.

The platform is productive.

The team can generate working software almost offensively quickly.

And the idea is wrong.

Now what?

Early-stage software startups make this constraint impossible to ignore because they operate under high uncertainty. The challenge is not merely implementing a known solution efficiently; it is discovering whether the problem, customer, solution, and product direction are worth pursuing at all. The Early Stage Software Startup Development Model applies lean principles around idea exploration, validation, and learning in precisely that environment.

Here, Speed finally gets demoted.

Learning becomes the authority.

A prototype is valuable because it tests an assumption.

An MVP is valuable because it exposes reality.

An iteration is valuable because it changes what the team knows.

Otherwise, rapid development simply lets us become wrong at higher throughput.

Fast delivery to the wrong problem is not a RAD success.

AI-Assisted Development: The Shortcut Starts Writing Back

Then we reach the newest acceleration layer.

Low-Code reduced how much implementation developers had to express manually.

AI-assisted development asks whether some of that remaining expression can come from natural language.

The AI Assistant is ridiculously fast.

It can suggest code, recall syntax, generate tests, help with refactoring, and support documentation-heavy tasks.

Developers surveyed about AI programming assistants reported motivations such as reducing keystrokes, completing programming tasks quickly, and recalling syntax. But the same study found major reasons for rejection: generated code may fail functional or non-functional requirements, and developers can struggle to control the assistant into producing the intended result.

That combination is the entire AI-assisted RAD tension in miniature.

It is fast.

Then context matters.

It is helpful.

Then authority matters.

It produces code.

Then correctness asks who approved it.

The AI Assistant is not exactly lying to us.

It is more like an extremely confident colleague who sometimes joins the conversation halfway through.

The deeper model is the same one that keeps appearing throughout this journey: reducing artifact-production effort does not remove the need to understand the system.

It changes where human effort goes.

Less typing.

More framing.

More context.

More review.

More validation.

More deciding whether the result actually belongs in the architecture.

The Hidden Constraint Was Never Development Speed

Now the pattern becomes difficult to ignore.

Classical RAD shortened the feedback cycle.

Executable mockups shortened the distance between requirements and something users could operate.

Low-Code shortened the distance between models and applications.

MDE shortened the distance between abstraction and implementation.

Citizen development shortened the distance between business expertise and software construction.

AI shortened the distance between intent and generated artifacts.

Every generation removed friction.

And every generation exposed another boundary.

Maintainability.

Integration.

Governance.

Non-functional requirements.

Architecture.

Reliability.

Context.

The mistake is thinking the next tool finally removes software engineering from software development.

It does not.

It moves the engineering attention toward whatever even The Origami Software Engineer cannot safely leave to the tool.

That is why there is no universal “best RAD tool”.

A workflow-heavy internal system may align beautifully with low-code.

A requirements-heavy business application may benefit enormously from executable mockups.

A model-intensive information system may fit MDE-backed automation.

A citizen-development program may work only when governance grows alongside participation.

A startup may need rapid experimentation more than rapid production.

A developer-led system may benefit from AI assistance precisely where the tasks are bounded enough to review.

The architecture starts making sense once we stop asking which tool is fastest and start asking which constraints the tool is allowed to hide.

RAD Did Not Disappear. It Kept Changing Where the Work Lived

Classical RAD began with a remarkably durable idea:

Get working software in front of reality sooner.

Everything that followed kept finding new ways to do that.

The Prototype became executable.

The Tool became a platform.

The Platform became model-driven.

The User became a Citizen Developer.

The coding assistant became conversational.

And each step looked, briefly, like the old complexity had vanished.

It had not.

It had moved.

From implementation into abstraction.

From construction into governance.

From syntax into context.

From producing artifacts into deciding whether those artifacts are correct, maintainable, secure, and worth building at all.

That is the part of rapid development that survives every generation.

Not “go faster”.

Learn sooner, automate what has become predictable, and keep human attention on the constraints that have not.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

Before you go

Please take a moment to like the post and follow the writer!
Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute here .

How Do Strangers Agree on One History?: Rethinking Distributed Computing #14

Aaroophan Varatharajan — Mon, 21 Sep 2026 06:46:01 +0000

Removing the central authority does not remove the need for one accepted history. It just makes the network responsible for deciding which history deserves to survive.

"How blockchain consensus works: distributed ledgers, double spending, digital signatures, cryptographic hashes, Proof of Work, mining, forks, chain selection, confirmations, network partitions, and decentralized consensus."

Suppose Alice, Bob, and Peter exchange value with one another.

We could move physical money every time.

Or we could maintain a ledger.

Alice pays Bob 300
Bob pays Peter 100
Peter pays Alice 50

The ledger records the history of transactions, and the current balances can be derived from that history.

Banks already make this idea feel ✨ordinary✨

The important part is not really the rows.

It is who is allowed to add them.

In a traditional system, some trusted institution owns the ledger.

Alice cannot simply write: Alice receives 10,000 from nowhere

and declare herself richer.

The authority validates transactions, controls updates, and provides one accepted version of history.

That gives us something extremely valuable:

There is one place to ask what happened.

Then distributed computing asks the dangerous question.

What if we remove that place?

Replicate the Ledger Everywhere

A centralized ledger comes with familiar problems.

One authority controls it.

One infrastructure operates it.

Its availability depends heavily on that infrastructure.

So imagine replicating the ledger across many independent nodes.

Now many participants hold copies.

The history no longer lives in one place.

That sounds wonderfully resilient.

NIST describes blockchain systems in this general form: a shared digital ledger maintained in a distributed fashion, with transactions grouped into blocks, cryptographically linked, replicated across network participants, and conflicts resolved according to established consensus rules.

So far, this looks like straightforward replication.

Alice transfers 300 to Bob.

Every ledger copy eventually records:

Alice: 500 → 200
Bob: 1000 → 1300

Easy.

Until the copies stop agreeing.

Because distributing the ledger did not distribute one magical global view of reality.

It created multiple machines capable of observing different things at different times.

And now the central authority is no longer there to settle the argument.

The Network Splits, and Alice Finds Two Realities

Suppose Alice begins with:

Alice = 500
Bob = 1000
Peter = 500

Alice sends: Transfer 300 to Bob

One part of the network sees it.

That ledger becomes:

Alice = 200
Bob = 1300
Peter = 500

Now imagine a network partition.

Another group of nodes cannot see that transaction yet.

From their point of view, Alice still owns: 500

So Alice sends another transaction: Transfer 400 to Peter

That side accepts it.

Its ledger becomes:

Alice = 100
Bob = 1000
Peter = 900

Now pause.

Alice started with 500.

One history says she spent 300.

Another says she spent 400.

If both histories survive, she has successfully spent: 700

from a balance of: 500

Nothing about copying the ledger prevented this.

In fact, replication created the conditions in which two groups could each believe a conflicting transaction was valid.

This is the double-spending problem.

And it exposes the actual problem blockchain has to solve.

Not:

How do we store transactions on many machines?

But:

How do many machines agree on which sequence of transactions counts as the history?

Bitcoin’s original design begins from exactly this difficulty: digital signatures can establish authorization, but a peer-to-peer system still needs a way to prevent the same value from being spent through conflicting histories without relying on a trusted third party.

Signatures Tell Us Who Authorized Something, Not Which History Won

Public-key cryptography helps.

Alice possesses a private key.

She can digitally sign a transaction.

Other participants can verify that signature using the corresponding public information.

So Alice cannot casually spend Bob’s value merely by writing: Bob pays Alice 500 without the authorization required by the system.

Good.

But signatures do not solve everything.

Alice can legitimately sign two conflicting transactions involving value she controls.

Alice → Bob 300
Alice → Peter 400

Both signatures can be perfectly valid.

Cryptography can answer:

Did Alice authorize this transaction?

It does not automatically answer:

Which of Alice’s conflicting transactions belongs in the accepted history?

That is a consensus problem.

The network needs something stronger than authentication.

It needs ordering.

Put Transactions Into Blocks, Then Make History Tamper With Itself

Instead of treating the ledger as unrelated lines, group transactions into blocks.

Then link each block to the previous one using a cryptographic hash.

Conceptually:

Block 100 ↓ hash
Block 101 ↓ hash
Block 102

Each block includes information derived from the previous block.

Change Block 100 and its hash changes.
Then Block 101’s reference no longer matches.
Fix Block 101 and its hash changes too.
Now Block 102 has to change.

And so on.

NIST describes this as the fundamental tamper-evident property of a blockchain: blocks are cryptographically linked to their predecessors, so altering older data changes those links and affects the chain that follows.

This gives history structure.

But there is still a missing piece.

Anyone can calculate hashes.

If producing a new valid block is cheap, a malicious participant could simply create an alternative sequence quickly.

So we need to make extending history costly.

Make the Right to Extend History Expensive to Produce and Cheap to Verify

Enter Proof of Work.

The idea feels backward at first.

Normally, computing systems try to make work cheaper.

Proof of Work deliberately asks participants to perform computation that is: difficult to produce, but easy for everyone else to verify.

For Bitcoin, a miner varies a value called a nonce while hashing block data, searching for an output that satisfies the current difficulty requirement.

There is no known shortcut that simply reveals the successful nonce.

You try. Hash. Try again. Hash. Again.

Eventually someone finds a valid result.

Everyone else can check it quickly.

Bitcoin’s Proof of Work is built around precisely this asymmetry: finding a suitable hash requires repeated computation, while verifying the resulting proof requires little work.

The network has created a strange kind of authority.

Nobody is appointed permanently.

Nobody is inherently trusted.

Anyone may compete.

But proposing the next block now requires evidence that computational work was actually performed.

You cannot merely shout:

My history wins.

You have to bring receipts.

The First Valid Block Still Does Not Give Us Permanent Truth

Suppose two miners solve the Proof-of-Work problem at nearly the same time.

Miner A broadcasts: … → Block X

Miner B broadcasts: … → Block Y

Both blocks extend the same previous block.

Both may contain valid transactions.

Both may contain valid Proof of Work.

Some nodes hear about X first.

Others hear about Y first.

Now we have:

Excellent.

We built a system whose entire purpose was to maintain one ledger, and it has produced two truths.

For a while.

And that last phrase matters.

Bitcoin explicitly allows temporary competing branches. Nodes can initially work on whichever valid branch they learned about first while retaining knowledge of the alternative.

The system does not guarantee that every node sees the same newest block at the same instant.

That would be difficult in a distributed network.

Instead, it gives the disagreement somewhere to go.

Keep Building, and Make the Competition Resolve Itself

Now another miner finds the next valid block.

Suppose it builds on X:

The X branch now represents more accepted Proof-of-Work history.

Nodes following Y can see that the competing branch has moved ahead.

So they switch.

Traditionally, this is described as:

The longest chain wins.

More precisely in modern Bitcoin implementations, nodes prefer the valid chain with the greatest accumulated Proof of Work rather than merely counting blocks. The older “longest chain” wording is useful intuition because the winning chain represents the branch backed by the greatest demonstrated work.

This is the clever part.

There is no global coordinator sending:

ATTENTION EVERYONE: X IS NOW OFFICIAL

Participants follow the same deterministic rule.

So as one valid branch accumulates more work, rational participants have reason to build on that branch rather than invest computation into a history that is increasingly likely to be discarded.

Agreement emerges from everyone following the same rule for which history deserves further work.

The Losing History Does Not Get to Keep Its Transactions

Now consider what happens to a transaction that existed only inside Y.

Perhaps Alice’s payment to Peter was there.

If the network converges on X, Y is no longer part of the accepted chain.

That transaction may therefore no longer be considered confirmed in the ledger history.

This is why seeing a transaction inside one block is not the same thing as receiving an eternal commitment from the universe.

A newer competing history can temporarily replace what a node previously believed was the best history.

Bitcoin’s developer documentation makes the consequence explicit: recently confirmed transactions carry more double-spend risk because recent blocks can occasionally be replaced when competing branches are resolved.

This can feel wrong if we bring database intuition with us.

In a conventional transaction system COMMIT usually means the transaction crossed a fairly crisp boundary.

In Proof-of-Work blockchain consensus, confidence can be probabilistic.

One block gives some confidence.

Another block built on top gives more.

Then another.

Then another.

The transaction becomes progressively harder to replace because replacing it requires rebuilding the work behind its block and catching up with the continuing chain.

The “commit point” has turned into a slope.

Confirmations Are Confidence Accumulating Behind You

Suppose Alice pays Bob.

The transaction enters a block.

That block now has: 1 confirmation

A new block is added after it.

Now: 2 confirmations

More blocks accumulate: 3 , 4 , 5 , 6

Each additional block represents more Proof of Work layered on top of the history containing Alice’s transaction.

An attacker trying to replace that transaction must construct a competing history and overcome the accumulated work of the chain that honest participants continue extending.

This is why systems can wait for additional confirmations before treating a transaction as sufficiently settled.

Bitcoin developer guidance historically uses six confirmations as an example of a transaction buried beneath substantial additional work, reducing the practical double-spend risk compared with a transaction that has only just appeared.

Notice what the system is saying.

Not:

Six means mathematically impossible to reverse.

But:

The deeper this transaction sits beneath continuing Proof of Work, the harder replacing that history becomes.

Persistence is being built incrementally.

Proof of Work Is Slow Because the Difficulty Is Doing a Job

At this point, Proof of Work looks wasteful.

Participants perform huge amounts of computation merely to find a value satisfying a hash condition.

Then everyone throws away almost all the failed attempts.

Why deliberately make something slow and expensive?

Because if valid history were cheap to manufacture, conflicting histories would also be cheap to manufacture.

The computational difficulty creates a cost for rewriting the past.

Once a block has been accepted and more blocks have accumulated after it, changing that block means reconstructing the Proof of Work for that block and its successors while the rest of the network continues extending the competing chain. Bitcoin’s design relies on exactly this asymmetry.

So the slowness is not an accidental performance bug.

It is part of the security model.

Of course, that creates a trade-off.

Transactions want to be:

fast,
persistent,
and inexpensive.

Proof of Work obtains stronger confidence by deliberately making block production costly.

The protocol is buying resistance to cheap history rewriting with computation and time.

Distributed systems rarely hand out guarantees for free.

The Majority Assumption Was Hiding Underneath Everything

Now suppose an attacker controls enough computational power to build an alternative chain faster than the honest network.

The chain-selection rule has no magical ability to ask:

Which history was created by morally good people?

It sees valid transactions, valid hashes, and accumulated Proof of Work.

If an adversarial group controls the majority of the relevant mining power, it may be able to outpace the honest branch and replace portions of recent history involving its own transactions.

This is the familiar 51% attack idea.

The Bitcoin design states its assumption directly: security depends on honest participants collectively controlling more computational power than any cooperating attacker group, so the honest chain grows faster than competing malicious histories.

This is worth lingering on.

Blockchain did not eliminate trust assumptions.

It changed them.

Instead of:

Trust this central bank.

or:

Trust this database administrator.

the Proof-of-Work system says something closer to:

Trust that the protocol’s validation rules are followed and that attackers do not control enough computational power to dominate history.

Decentralization did not make trust evaporate.

It relocated the trust boundary into the consensus mechanism and its assumptions.

Consensus Does Not Mean Everyone Has the Same View Every Second

This is perhaps the strangest mental shift.

When we hear consensus , we naturally imagine everyone simultaneously agreeing.

Node A: X
Node B: X
Node C: X

Blockchain consensus can spend some time looking more like:

Node A: X
Node B: Y
Node C: X
Node D: Y

Then:

Node A: X → X2
Node B: X → X2
Node C: X → X2
Node D: X → X2

Temporary disagreement does not automatically mean consensus failed.

It can be part of the mechanism through which consensus eventually emerges.

NIST describes blockchain consensus mechanisms as the rules by which distributed participants determine which proposed state becomes accepted, with forks representing temporary divergent histories that must be resolved.

That is a softer notion of consistency than a single authoritative database gives us.

But it solves a different problem.

The participants may not trust one central authority.

They may join and leave.

They may receive network messages in different orders.

Some may even behave maliciously.

The protocol’s job is therefore not necessarily:

Prevent disagreement from ever appearing.

It is:

Give disagreement a rule for converging.

That is a much more distributed kind of promise.

The Blockchain Is Really a History-Selection Machine

It is easy to focus on the visible objects.

Blocks.
Hashes.
Nonces.
Miners.
Coins.

But underneath them sits one recurring question:

Which history should an independent node believe?

The hash chain makes history tamper-evident.

Digital signatures establish who authorized transactions.

Proof of Work makes producing candidate history expensive while keeping verification cheap.

The chain-selection rule tells nodes what to do when multiple candidate histories exist.

Confirmations let confidence accumulate as more work is built on top.

The majority-computation assumption explains when that process remains secure against a competing attacker.

Put together, these mechanisms create something that a centralized ledger gets much more cheaply:

one history that participants can treat as authoritative enough to continue from.

That is why blockchain is fundamentally a distributed-systems story.

Timing matters.

Ordering matters.

Network partitions matters.

Consensus matters.

Consistency and availability matters.

The Origami Software Engineer’s peace matters.

The cryptography protects particular relationships inside the system.

It does not make the distributed-systems problems disappear.

We Removed the Authority, Then Rebuilt What the Authority Used to Give Us

A central ledger gives us several things almost invisibly.

One place controls updates.

One place orders transactions.

One place rejects double spending.

One place says which history is current.

Remove that authority and none of those responsibilities disappear.

They become protocol problems.

So the system evolves.

Replicate the ledger.

Now replicas can disagree.

Sign transactions.

Now authorization is verifiable, but conflicting authorized transactions can still exist.

Chain blocks with hashes.

Now history becomes tamper-evident.

Make new blocks require Proof of Work.

Now producing competing history has a real cost.

Allow temporary forks.

Now the system can survive nodes learning about valid blocks in different orders.

Choose the chain with the greatest demonstrated work.

Now independent nodes have a shared rule for convergence.

Wait for confirmations.

Now persistence becomes progressively stronger instead of instantly absolute.

Assume honest participants control enough computational power.

Now we can finally state the condition under which that convergence resists an attacker.

Every mechanism exists because removing the trusted ledger owner exposed another responsibility that the owner used to handle for us.

That is the deeper lesson.

Decentralization does not mean nobody decides.

It means the right to decide is no longer concentrated in one trusted actor.

The decision has to emerge from rules that independent participants can verify and follow for themselves.

The strangers do not need to trust one another personally.

They need enough shared rules to eventually agree on which history is worth continuing.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

When COMMIT Stops Being a Local Decision: Rethinking Distributed Computing #13

Aaroophan Varatharajan — Sat, 19 Sep 2026 07:46:01 +0000

On one database, commit is a decision. Across several machines, commit becomes an agreement that has to survive failures.

"How Two-Phase Commit (2PC) coordinates distributed transactions across multiple machines, handling prepare, commit, failures, crash recovery, blocking, and atomicity."

Consider a transaction on one database.

Change some data.

If everything succeeds: COMMIT

If something goes wrong: ROLLBACK

The database owns the relevant state, so there is one ✨obvious✨ place where the final decision happens.

Atomicity feels pleasantly local:

Either all of this transaction becomes permanent, or none of it does.

Now spread the data across multiple servers.

Account A lives on Server 1.

Account B lives on Server 2.

We want one transaction to:

A = A + 1
B = B - 1

Logically, this is still one indivisible transaction.

Physically, two different machines have to modify two different pieces of data.

And that tiny change ruins the simplicity of COMMIT.

Because Server 1 can succeed while Server 2 fails.

The transaction is logically one thing.

The machines performing it are not.

Partial Success Is Worse Than Failure

Suppose a transaction coordinator tells:

Server 1 → add 1 to A
Server 2 → subtract 1 from B

Both operations execute tentatively.

Now imagine the simplest possible protocol:

Once you’ve done your part, commit it.

Server 1 commits.

Server 2 crashes before committing.

We now have:

A = A + 1
B = unchanged

The system did not merely fail to complete the transaction.

It completed half of it permanently.

That is much worse.

If neither side had committed, we could simply tell the client:

The transaction failed. Try again.

The previous consistent state would still exist.

But once one participant commits and another does not, retrying is no longer obviously safe. Some of the intended change already escaped into permanent state.

The uncomfortable realization is:

Distributed atomicity is not about making every operation succeed. It is about preventing different participants from reaching incompatible final outcomes.

Oracle describes distributed transactions in exactly this all-or-nothing form: changes span multiple databases, but the transaction must still commit everywhere or roll back everywhere as one unit.

So perhaps the coordinator should make the decision for everybody.

Fine. Broadcast COMMIT

The coordinator waits until both servers have performed their local work.

Then it sends:

COMMIT → Server 1
COMMIT → Server 2

Much cleaner.

One authority.

One decision.

Both participants do what they are told.

Except the network gets a vote.

Server 1 receives: COMMIT

Server 2 does not.

Maybe its machine crashed.

Maybe the message disappeared.

Maybe the connection failed.

Server 1 commits.

Server 2 does not.

We are back where we started.

The coordinator decided correctly.

That was not enough.

The hidden problem is that sending the final decision does not prove that every participant is capable of carrying it out.

So before the system says:

Commit.

it needs another question.

Can you commit?

That extra question is where Two-Phase Commit begins.

The First Phase Is Not COMMIT

In Two-Phase Commit , or 2PC , one process acts as the transaction coordinator and the machines containing pieces of the transaction act as participants.

The coordinator does not immediately order everyone to commit.

It first sends something like: PREPARE

Meaning:

If I later tell you to commit this transaction, are you able to guarantee that you can do it?

Each participant checks its local situation.

Did its part of the transaction execute successfully?

Can constraints still be satisfied?

Can it preserve the information required to finish later?

Can it keep the necessary transaction state?

If not, it votes: NO

If it can guarantee completion, it votes: YES

Oracle’s description of the prepare phase captures the important semantic difference: a prepared participant has not yet committed, but it promises that it will later be able to follow the global commit or rollback decision even across an intervening failure.

This is much stronger than:

Everything looks fine right now.

A YES vote is a promise about the future.

And promises in distributed systems need receipts.

“YES” Has to Survive a Crash

Imagine Server 1 replies: YES

Then immediately crashes.

When it restarts, what should happen?

If the promise existed only in RAM, the server wakes up with no idea that it had agreed to participate in an unfinished distributed transaction.

That makes the YES meaningless.

So before a participant says it is prepared, it must preserve enough transaction state in stable storage.

PostgreSQL’s PREPARE TRANSACTION illustrates this principle directly: once prepared, the transaction's state is stored on disk so it can later be committed or rolled back even if a database crash occurs before the final decision.

Oracle similarly describes prepared participants recording recovery information in redo logs and retaining the resources needed to complete the transaction after failure.

So the participant’s logic becomes roughly:

perform tentative work
write prepared state durably
vote YES

Not:

vote YES
hope nothing happens

The durable log is what turns:

I think I can commit.

into:

I can recover into a state from which I can still honor this decision.

Prepared Means You Have Given Up Some Freedom

This is the part that makes the first phase more important than its name suggests.

Before voting YES , a participant can still say:

Something went wrong. Abort my local work.

After voting YES , it has promised not to make that decision independently.

It is now prepared.

It holds enough state to commit.

It may also retain locks or other resources required to preserve correctness.

And then it waits.

The participant cannot decide:

This is taking too long. I’ll just roll back.

Because the coordinator may already have received YES votes from everyone and decided to commit globally.

Oracle explicitly describes a prepared node as waiting for the global COMMIT or ROLLBACK decision rather than making a unilateral choice. The prepared transaction can remain in-doubt until that decision becomes known.

That is the trade.

The participant gained durability of intent.

It lost local authority over the outcome.

Now the Coordinator Can Actually Decide

Suppose the coordinator asks three participants: PREPARE?

and receives:

P1 → YES
P2 → YES
P3 → YES

Only now does it know something it did not know in our naive protocol:

Every participant has reached a state from which it can complete the transaction.

So the coordinator can choose: GLOBAL COMMIT

and send the decision to everybody.

But suppose the votes are:

P1 → YES
P2 → NO
P3 → YES

Then commit is impossible.

The only safe global outcome is: GLOBAL ABORT

Likewise, if a participant cannot prepare and the coordinator eventually treats that participant as failed, the transaction can be aborted before the global commit decision.

This gives us the two phases:

Phase 1: Prepare

Coordinator → PREPARE
Participants → YES / NO

Phase 2: Decision

If everyone voted YES : Coordinator → COMMIT

Otherwise: Coordinator → ABORT

This is why the protocol needs two phases.

The first establishes whether commitment is possible everywhere.

The second establishes which outcome everyone must perform.

One phase cannot safely answer both questions after the system has been distributed.

Before the Decision, Aborting Is Cheap

Suppose the coordinator sends PREPARE to three participants.

Two reply: YES

The third never responds.

Maybe it crashed.

Maybe communication failed.

At this point the coordinator has not decided to commit.

That matters.

Because no irreversible global commitment has been made, the coordinator can decide: ABORT

The participants that were prepared can roll back their tentative work.

The transaction disappears.

We return to the previous consistent state.

This is one of the nice properties of the prepare boundary:

Before universal readiness exists, failure can still collapse safely into abort.

No participant is allowed to commit merely because its own local work succeeded.

Local success is not global success.

That distinction is exactly what the naive protocol was missing.

Then We Cross the Uncomfortable Line

Now take the other case.

Every participant votes: YES

The coordinator has enough information to decide: COMMIT

Suppose it sends that decision to Server 1.

Server 1 commits.

Then the network fails before Server 2 receives the message.

We cannot now say:

Never mind. Abort everything.

Server 1 has already made the committed state permanent.

The global decision has crossed a boundary.

The transaction must now be driven toward commit everywhere.

This creates what the lecture calls the critical window: after the system has committed to the global outcome but before every participant has successfully completed and acknowledged it.

The coordinator cannot safely pretend the transaction never happened.

The remaining participants cannot independently guess the answer.

The system has entered the most awkward state distributed transactions produce:

The outcome exists, but not everybody knows it yet.

Crash Recovery Becomes Part of COMMIT

Suppose Server 2 crashed after voting YES but before receiving the final decision.

When it comes back, its durable log says: Transaction 42 → PREPARED

That tells Server 2:

I promised I could finish this transaction, but I do not know whether the global outcome was COMMIT or ABORT.

So it must recover the prepared transaction and determine the coordinator’s decision.

If the decision was COMMIT , it commits.

If it was ABORT , it rolls back.

The prepared state is precisely what allows this recovery to happen without guessing.

Similarly, the coordinator must durably remember its own decision.

Imagine it decides COMMIT , sends the message to one participant, then crashes.

When it restarts, it cannot wake up and reconsider:

Maybe let’s abort instead.

Somebody may already have committed.

So the coordinator’s global outcome also belongs in stable storage.

MIT’s treatment of 2PC makes this durability requirement explicit: the coordinator force-writes its decision, prepared workers recover into the prepared state and ask for the outcome, and the coordinator retains the outcome until workers acknowledge it.

The log is no longer merely an optimization for crash recovery.

It is part of the protocol’s memory.

Then the Coordinator Disappears at Exactly the Wrong Time

Now we reach the part where 2PC stops looking like a magical atomicity machine.

A participant has voted: YES

It is prepared.

It has preserved its state.

It is waiting for: COMMIT or ABORT

Then the coordinator becomes unreachable.

What should the participant do?

It cannot simply commit.

Perhaps another participant voted NO and the coordinator chose abort.

It cannot simply abort.

Perhaps everyone voted YES , the coordinator chose commit, and another participant already committed.

So the prepared participant waits.

Its uncertainty is legitimate.

Oracle calls this an in-doubt distributed transaction : failure during 2PC can leave prepared participants waiting for the outcome, with associated data remaining locked until the transaction can be resolved.

Academic treatments describe the same property more bluntly:

Two-Phase Commit is a blocking protocol.

If all surviving participants are prepared but none knows the global decision, they may have no safe action except waiting for the failed coordinator or another source of decisive information to recover.

That is not an implementation bug.

It follows from what *YES * means.

The Prepared Participant Is Stuck Because It Is Being Correct

This state can feel ridiculous.

The server is alive.

Its database is alive.

Its disks are alive.

It has the transaction data.

And yet it refuses to either commit or roll back.

Why?

Because it lacks authority.

Before preparing, the participant owned its local choice.

After preparing, it delegated the final choice to the distributed commit decision.

Now the coordinator is unavailable.

The participant is not waiting because it is incapable of performing either operation.

It is waiting because it cannot determine which operation preserves atomicity.

That is an important distinction.

The server is technically capable.

The system is informationally stuck.

The Origami Software Engineer has seen this recurring distributed-systems pattern in another costume:

Missing communication becomes missing knowledge.

And once the missing knowledge determines whether permanent state should exist, guessing becomes dangerous.

Locks Make the Waiting Visible

Prepared transactions cannot casually release everything they were using.

If another transaction could modify the same state while the original transaction’s outcome remained unresolved, the participant might no longer be able to honor its promise correctly.

So prepared transactions may retain locks.

Now uncertainty starts affecting unrelated work.

Transaction 42 is in doubt.

Transaction 73 wants the same data.

Transaction 73 waits.

Oracle’s documentation explicitly notes that in-doubt distributed transactions can keep data locked until the outcome is resolved; its error documentation even describes resources remaining locked by prepared distributed transactions.

This is where availability pays the price for atomicity.

2PC’s promise is:

We will not let half the distributed transaction permanently win while the other half permanently loses.

The cost is that, under some failures, the system may prefer to wait rather than guess.

Correctness is being protective.

Possibly overprotective.

But for good reason.

Two-Phase Commit and Two-Phase Locking Are Solving Different Problems

The names are similar enough to cause unnecessary suffering.

Two-Phase Locking is about concurrent transactions.

It helps preserve consistency and isolation by controlling the ordering of conflicting accesses.

Two-Phase Commit is about one transaction whose work spans multiple participants.

It helps preserve atomicity and durability by coordinating whether every participant commits or every participant aborts.

One asks:

Can these concurrent operations safely interleave?

The other asks:

Can these distributed pieces make one final decision?

A system may need both.

Locks can ensure that other transactions do not corrupt the objects involved.

2PC can ensure that the distributed transaction does not commit on only some machines.

Different problem.

Different mechanism.

Same uncomfortable theme:

Once state is distributed, a property that looked local requires coordination.

The Extra Phase Is Buying Certainty

2PC is not cheap.

Every distributed transaction now involves more messages.

Before commit:

PREPARE
votes

Then:

COMMIT / ABORT
acknowledgements

Prepared state has to be written durably.

The coordinator must persist enough information to recover its decision.

Participants may retain locks while waiting.

Failures can leave transactions in doubt.

That can reduce availability.

The transaction that once ended with: COMMIT has become a small distributed protocol.

And that is precisely the point.

The additional machinery exists because the simple operation stopped carrying enough information.

One local database can know:

My work succeeded.

A distributed transaction needs to know:

Everyone can complete the same outcome, and that outcome will survive crashes and communication failures.

That stronger claim costs more.

COMMIT Became a Conversation Because Nobody Owned the Whole Transaction

At the beginning, the problem looked tiny.

Move one unit:

A = A + 1
B = B - 1

When both values lived under one transactional authority, atomicity could be enforced locally.

Then A and B moved onto different machines.

Now Server 1 could succeed while Server 2 failed.

So independent commit was unsafe.

We introduced a coordinator.

But one COMMIT broadcast could still reach only some participants.

So the coordinator first asked whether everybody was prepared.

A YES vote became a durable promise.

Once everybody promised, the coordinator could choose one global outcome.

Then failures created another problem: some participants might know the decision while others remained uncertain.

So participants and coordinator had to preserve protocol state across crashes.

And once a prepared participant could no longer discover the decision, waiting became safer than guessing.

That is why COMMIT stops being one instruction in a distributed transaction.

It becomes a conversation about readiness, authority, durability, and shared outcome.

Two-Phase Commit does not make failures disappear.

It creates a disciplined boundary around what the system is allowed to do when failures happen.

Before the boundary, abort is still safe.

After the boundary, the chosen outcome has to be carried through.

The extra phase was never ceremony.

It was the price of turning several independent machines back into one atomic decision.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

DEV Community: Aaroophan Varatharajan

The Distributed File System That Wins Depends on What You’re Afraid Of

Why NFS, AFS, Coda, and GFS Make Different Trade-Offs for Different Failures

Transparency Looks Easy Until the Network Starts Having Opinions

Mobility Exposes the Assumption AFS Was Still Making

Performance Stops Being a Number

Scalability Is Usually Someone Becoming Tired

Availability and Reliability Look Similar Until Something Actually Fails

Data Integrity Reveals Why Availability Has a Price

Security Is Where Old Trust Assumptions Start Showing

The Leaderboard Was the Wrong Abstraction

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Find me elsewhere:

When "Just Make the Interface Easy" Stops Being Enough: Rethinking Human Computer Interaction

Why goals, tasks, translation, ergonomics, interaction styles, error recovery, context, and perceived value shape usable systems

The Seductive Model: User In, System Out

Domain, Goal, and Task: The First Crack

Norman Gives the Negotiator a Workflow

The Gulfs: Where a Correct System Can Still Feel Wrong

Abowd and Beale Reveal the Translation Problem

Ergonomics Reminds Software That Humans Have Bodies

Glass Interfaces Solve One Problem and Introduce Another

Interaction Styles Are Different Kinds of Conversation

WIMP Makes Interaction Easy, Then Becomes an Ecosystem

"Look" Was Never Enough

Initiative, Error, and the Right to Recover

Context Changes the Same Interface Without Touching the Code

"Can Use" Becomes "Want to Use"

Physical Design Turns Certainty Into Trade-offs

Fluidity Is When Physical and Logical State Stop Arguing

Value Was the Constraint Hiding Under Everything

The Interface Was Never the Real Goal

The "I liked this" Starter Pack:

Don’t let your fingers get lazy now.

Find me elsewhere:

“Send” Is Not One Operation: Rethinking Distributed Computing #5

The dangerous part of send(message) is how much meaning we quietly pack into the word “send.”

First Question: How Long Should the Message Stay Alive?

Second Question: When Is the Sender Allowed to Continue?

The Queue Says, “I’ll Hold It”

Queued Does Not Mean Processed

Then We Stop Naming the Receiver

Somebody Still Has to Understand the Message

A Message Bus Is Not a Conveyor Belt

MPI Wants Speed More Than Comfort

MPI_Send Still Does Not Mean What You Think

MPI Has More Than One Kind of Send for a Reason

Receiving Requires Its Own Contract

Then Point-to-Point Becomes Awkward

A Collective Is a Group Contract

“Send” Was Really a Question About Responsibility

Make the Contract Say What the System Means

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Find me elsewhere:

A Remote Function Call Will Never Really Be Local: Rethinking Distributed Computing #4

The API can hide the distance. It cannot erase what the distance changes.

The Network Does Not Know What a Function Is

Even the Arguments Cannot Travel as Themselves

Then We Have to Find the Procedure

Local Calls Have a Very Comfortable Timeline

The Timeout Is Where the Illusion Gets Cursed

Fine. Retransmit It.

Did the Server Run It?

Failure Transparency Has a Limit

Then We Tried It With Objects

Remote Exceptions Are the Abstraction Admitting the Truth

Transparency Should Remove Work, Not Reality

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Find me elsewhere:

Navigating the CAP Theorem: Rethinking Distributed Computing #3

Replication gives us another copy of the truth. Then the network disappears, and both copies have to decide whether they are still allowed to call themselves true.

The Problem Is Not That Either Server Is Wrong

“Pick Two” Is a Little Too Comfortable

CP Says “I Would Rather Refuse Than Lie”

AP Says “I Would Rather Answer and Reconcile Later”

AP Is Not “No Consistency”, and CP Is Not “No Availability”

Then the Backup Decides the Primary Is Dead

The dangerous part of `send(message)` is how much meaning we quietly pack into the word “send.”

`MPI_Send` Still Does Not Mean What You Think