DEV Community: Aaroophan Varatharajan

More Data Doesn’t Mean Better Decisions: Rethinking Business Intelligence

Aaroophan Varatharajan — Sun, 01 Nov 2026 17:13:46 +0000

A company can collect enormous amounts of information, build beautiful dashboards, and still make terrible decisions. Why?

Business Intelligence is not about having more data. It is about reducing the distance between what an organization knows and what it can intelligently do.

A supermarket records every sale. Its online store records purchases too. Loyalty accounts can connect some of those purchases to customers, while inventory systems keep track of what is left on the shelves.

The obvious next step is tempting:

Collect all that data,

Put it on a dashboard,

and let management make ✨better✨ decisions.

Why wouldn’t that work?

The numbers are there.

The charts are there.

The dashboard even refreshes itself.

The uncomfortable part is that none of those things guarantees that the business knows what to do.

The Most Reasonable First Move Is to Count Everything

Start with the simplest possible approach. Bring the records together, calculate total sales, group the transactions by product or branch, and draw a few charts. A manager can now answer questions that previously required searching through spreadsheets or asking several departments for reports.

That is real progress. Business Intelligence (BI) is partly about making information accessible instead of leaving it buried in operational systems.

But the original question was not How many transactions do we have?

It was How can we make better decisions?

Suppose sales are declining. A dashboard can show when the decline began and where it appears. It does not automatically explain whether the problem is pricing, product availability, customer preferences, or something outside the business. Adding another chart may reveal another pattern, but the missing explanation has not magically appeared.

This is the trap: Visibility feels a lot like understanding, especially when the visualization looks convincing.

The Missing Component Is Not Necessarily Another Data Source

A typical BI process collects and aggregates data, turns it into reports, and makes the results available to people who need them. Databases, data warehouses, and Extract-Transform-Load (ETL) tools help assemble the information; reporting and dashboard tools make it readable.

Then comes a step that is easy to underestimate: C ontext.

Context comes from the decision being made and from the people who understand the business. A customer-service team may know why particular customers are dissatisfied. A marketing team may know which promotions are already running. Neither perspective is guaranteed to be visible in a sales-total chart.

The decision process is closer to this:

Notice what the diagram does not say. A dashboard does not become a decision simply because its data is fresh. It presents evidence. Someone still has to interpret that evidence for a purpose.

Even a perfectly accurate report can be insufficient when the question requires information the report does not express.

Data, Information, and Knowledge Are Not Synonyms

The distinction becomes clearer with a smaller example.

Consider a restaurant order containing a burger, fries, and a milkshake. That order is data : a record of what was entered into the system. Gather many orders, filter them by day, and calculate how many burgers were sold. Now we have information : processed data relevant to a particular question.

If repeated observations reveal a useful relationship between demand and the quantities the restaurant should prepare or order, we begin to form knowledge. That relationship can inform a future stock decision.

Data : Recorded raw facts that may still contain errors, such as individual restaurant orders and item quantities.
Information : Data processed and organized for a specific purpose, such as daily sales totals for burgers.
Knowledge : Insights gained by interpreting information and identifying useful patterns, such as predicting burger demand to prepare sufficient stock.

The difference is not that one stage has fancier software. It is that each stage answers a more consequential question.

Data tells us what was recorded. Information helps us see what those records mean in a particular context. Knowledge helps us reason about what to do next.

And knowledge is not infallible. If the underlying records are inaccurate, or if an observed relationship no longer holds, confidence can grow faster than understanding. A polished dashboard will not warn us about every bad assumption behind it.

A Churn Prediction Is Not a Retention Strategy

Customer churn makes this boundary harder to ignore.

A subscription provider has historical records of customers who stayed and customers who left. Using those records, it can build a model to identify current customers who appear more likely to leave.

That sounds like the solution. We found the at-risk customers. Done.

Not quite.

The model identifies a pattern in historical data. It does not independently decide which customers should receive a promotion, what the offer should contain, or whether the apparent risk reflects a problem the company can actually address.

Those questions require business context. Marketing and customer-service teams can examine the identified group before deciding on an appropriate response.

Compare this with the tempting alternative : Give everyone a discount because some customers might leave. A targeted approach may avoid treating all customers as though they have the same needs. But the prediction alone is not proof that any particular offer will work.

The important transition is from “Who might leave?” to “What response makes sense for these customers?”

The first question can be supported by a model. The second needs a decision.

This Is Why BI Is Bigger Than the Tool on the Screen

The confusion becomes understandable when several related technologies are treated as interchangeable.

A data warehouse organizes data for analysis.

Data mining searches for patterns and relationships.

A decision support system helps people evaluate decisions using data, information, and knowledge.

These can contribute to BI, but none of them alone describes the entire process.

Business analytics overlaps with BI, especially where analysis, prediction, and optimization are involved. The terminology varies across organizations; a useful distinction is that analytics provides analytical methods, while BI also concerns the wider journey from collecting and organizing evidence to communicating it for decisions. IBM’s overview of BI and its discussion of business analytics make this relationship explicit.

That wider view also explains two different ways BI can help an organization:

Responsive decisions: something has happened, the organization detects it, and it responds.
Anticipative decisions: historical information suggests what may happen, allowing the organization to plan before it does.

Monitoring an operational problem supports the first. Identifying customers at risk of churn and considering a retention offer supports the second. Neither requires pretending that the analytical output is the same thing as the final choice.

Why the Pressure Keeps Increasing

If this were only about making charts, cheaper storage and faster processors would have solved most of the problem already.

Instead, The Origami Software Engineer face several pressures at once. Digital activity produces more records. Storage and processing have become easier to obtain. Regulations affect what information must be retained or protected. Decisions increasingly depend on factors beyond one department, and sometimes beyond the organization. Meanwhile, many useful decisions must be made quickly enough for the information to remain relevant.

More data can help with those pressures. It can also make an unstructured information problem larger.

A retailer, for example, may use historical purchases to plan stock, customer segmentation to tailor promotions, and market-basket analysis to identify products often purchased together. But inventory creates an uncomfortable trade-off: Too little stock means missed demand; Too much ties up resources and risks waste. The business needs an estimate that can guide that trade-off, not merely a larger collection of sales records.

The value of BI therefore shows up in places that are more concrete than the dashboard itself: better operational performance, improved customer service, and the ability to recognize opportunities that were previously difficult to see.

The Last Dependency Is Still a Decision

A useful BI system does not eliminate judgment. It makes judgment better informed.

We began with a reasonable idea: If more transactions become more charts, management should naturally make better decisions. The approach works up to a point. It makes the organization more aware of its own activity.

Then the hidden constraint appears. A record is not an explanation, an explanation is not a decision, and a decision is not automatically a good outcome.

Once that distinction is visible, the architecture makes more sense. Collect data because a decision needs evidence. Organize it because evidence must be accessible. Analyze it because patterns are not always obvious. Present it because other people must understand it. Add business context because numbers do not know the business’s objectives on their own.

The goal is not a dashboard that knows everything. It is an organization that can make a better choice with what it knows.

That’s not failure.
That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

Complete Staging Backend Release to Cloud Run Using a Lightsail Deployment VM

Aaroophan Varatharajan — Sat, 17 Oct 2026 07:46:01 +0000

A practical Cloud Run deployment workflow using Lightsail, Cloud SQL, Prisma, Artifact Registry, and Firebase staging infrastructure

A technical guide to deploying a staging backend to Google Cloud Run from a Lightsail VM, covering Cloud SQL connectivity, Prisma migrations, environment-specific DATABASE_URLs, Docker images, Artifact Registry, Cloud Storage, and immutable Cloud Run revisions.

A staging backend sounds almost insultingly simple.

Copy the development setup. Create a staging database. Change a few environment variables. Build the container. Deploy it.

✨Done✨

That intuition works right up until the same DATABASE_URL needs to mean two different things depending on where the command is running.

That is where staging stops being “dev, but elsewhere” and starts becoming an environment of its own.

Copying dev is the obvious starting point

Suppose an existing deployment VM already has a structure like:

backend/
└── dev/
    └── application-backend/

The reasonable first move is not to invent another deployment system.

Create staging beside dev:

cd /home/ubuntu/backend
mkdir -p stg
cp -a dev/. stg/

Now we have:

backend/
├── dev/
└── stg/
    └── application-backend/

This is deliberately boring.

The Dockerfile already works. The repository is already there. The deployment VM already has Docker, Git, Node.js, Prisma dependencies, and access to the container registry.

Reusing that machinery reduces the number of new things staging can break.

But copying the directory only duplicates the deployment shape.

It does not create an isolated environment.

For that, staging needs its own state.

The database is where staging becomes real

A new low-spec PostgreSQL Cloud SQL instance was created from the Google Cloud Console.

The important decisions were not glamorous:

Same GCP project family
Same region as the backend
Same PostgreSQL major version as the existing environment
Single-zone availability
Low-cost staging-sized compute
Separate staging database
Separate credentials

The staging database was then restored from an existing database backup so that the application had realistic structure and data to work with.

At this point, the tempting configuration looks like this:

# Deploying to GCP Cloud Run - STG
NODE_ENV="DEV"
DATABASE_URL="postgresql://...@localhost:5432/app_stg?host=/cloudsql/<instance-connection-name>"
GCP_PROJECT_ID="<project-id>"
GCP_STORAGE_BUCKET="<staging-bucket>"
FRONTEND_URL="<staging-frontend-url>"

That database URL is perfectly reasonable.

For Cloud Run.

Then we try to run migrations from the Lightsail VM.

And suddenly localhost starts lying to us.

One database, two network contexts

This was the important constraint.

Cloud Run and the Lightsail deployment VM do not reach Cloud SQL the same way.

When the backend runs inside Cloud Run, the service can be connected to a Cloud SQL instance and access its Unix socket through:

/cloudsql/<INSTANCE_CONNECTION_NAME>

So this kind of URL makes sense inside Cloud Run:

DATABASE_URL="postgresql://<user>:<password>@localhost:5432/app_stg?host=/cloudsql/<instance-connection-name>"

But npx prisma migrate deploy is not running inside Cloud Run.

It is running here: Lightsail VM

There is no Cloud Run-mounted /cloudsql/... socket there.

The same URL that is correct for the deployed application is wrong for the deployment machine.

That is the trap.

The database has not changed.

The network boundary has.

So staging ended up needing two configuration blocks.

One for the deployed application:

# Deploying to GCP Cloud Run - STG
NODE_ENV="DEV"
DATABASE_URL="postgresql://<user>:<encoded-password>@localhost:5432/app_stg?host=/cloudsql/<instance-connection-name>"
...

And another for commands executed directly from the deployment VM:

# Running locally - STG
# NODE_ENV="DEV"
# DATABASE_URL="postgresql://<user>:<encoded-password>@<cloud-sql-public-ip>:5432/app_stg?schema=public"
# ...

Only one is active at a time.

That sounds slightly cursed until the mental model clicks:

While runtime is:

Same database.

Different caller.

Different connection path.

Public IP access is not just “make the URL public”

Using the Cloud SQL public IP introduces another boundary.

The deployment VM has to be allowed through it.

So the VM’s public IP was added in:

Google Cloud Console → Cloud SQL → Staging instance → Connections → Networking → Authorized networks

As a single-host network: /32

Now the VM can reach PostgreSQL directly.

Before running anything destructive-looking, the active configuration can be checked:

grep -n "^[^#].*DATABASE_URL" .env.dev

For the migration step, there should be exactly one active DATABASE_URL, and it should contain:

@<cloud-sql-public-ip>:5432/<staging-database>

That tiny check matters.

There are few feelings more cursed than a successful migration against the wrong database.

Migration is part of the release, not the container

With the public-IP staging configuration active, the target release branch can be checked out:

git fetch origin
git checkout -B <release-branch> origin/<release-branch>
git status

Then:

npx prisma migrate deploy

Followed by:

npx prisma generate

These commands have different responsibilities.

migrate deploy applies the migrations already committed to the repository.

It does not invent new migrations.

It does not reset the database.

And it does not regenerate Prisma Client artifacts.

That is why prisma generate follows it.

The ordering becomes:

Only after those succeed should the environment file be switched back.

The public-IP configuration gets commented again.

The Cloud Run configuration becomes active:

DATABASE_URL="postgresql://<user>:<encoded-password>@localhost:5432/app_stg?host=/cloudsql/<instance-connection-name>"

Then check again:

grep -n "^[^#].*DATABASE_URL" .env.dev

This time the active URL should contain:

host=/cloudsql/<instance-connection-name>

This switching looks awkward at first.

But the awkwardness is telling us something useful.

Deployment-time connectivity and runtime connectivity are different responsibilities.

Now the container becomes the release artifact

Once the database is ready and the runtime configuration is active again, the application can be packaged.

First identify exactly what is being released:

git log --oneline

Take the short commit hash and combine it with a release version and environment name.

Then build:

docker build -t application-backend:latest .

Tag:

docker tag application-backend:latest \
  <region>-docker.pkg.dev/<project>/<repository>/application-backend:<commit>-v<version>-stg

Push:

docker push \
  <region>-docker.pkg.dev/<project>/<repository>/application-backend:<commit>-v<version>-stg

The tag is doing more work than it appears to.

Something like:

<commit>-v2.xx-stg

answers three useful questions:

Which source?
Which release?
Which environment?

The image is no longer “whatever was on the VM.”

It is now a named deployment artifact in Artifact Registry.

That boundary is important because Cloud Run is not deploying the working directory.

It is deploying the image.

Cloud Run does not edit the old deployment

The final step happens in the Google Cloud Console:

Cloud Run → Services → → Edit & deploy new revision

Then:

Edit container → Container image → Select the newly pushed image → Deploy

The wording matters: new revision.

Cloud Run does not quietly mutate the previous revision.

A changed image or changed service configuration produces another revision.

That gives the release a clean progression:

Each stage points to something concrete.

If the new revision misbehaves, the previous revision still exists.

The deployment has history instead of amnesia.

A staging backend is more than compute

The backend was now running.

Then another constraint became visible.

The staging database had been restored from production data, but staging had its own brand-new storage bucket.

The database knew about existing logos and artifacts.

The staging bucket knew absolutely nothing about them.

The application asked for an artifact.

The database said, “Sure, it exists.”

The empty bucket said, “Never heard of it.”

Nothing was technically lying.

They simply represented different copies of state.

Temporarily pointing staging at the existing artifact bucket made the old assets visible, but that is not the clean boundary we want long term. A staging environment should not casually share writable production storage.

The better direction is to migrate or copy the required artifacts into the staging bucket while preserving object paths, then return staging to:

GCP_STORAGE_BUCKET="<staging-bucket>"

This revealed a broader point about environment cloning.

Copying a database backup does not clone an application environment.

The database may contain references to:

object storage
external URLs
generated files
identity providers
frontend origins
secrets
third-party services

State has dependencies too.

The release flow that finally makes sense

Once all those constraints are visible, the staging release stops looking like a random collection of commands.

It has an order:

The uncomfortable part is that several configurations are all “correct.”

They are just correct in different contexts.

The public-IP database URL is correct from the deployment VM.

The Unix-socket database URL is correct from Cloud Run.

The source tree is correct for building.

The container image is correct for deployment.

The database backup is correct for relational state.

The storage bucket is responsible for artifact state.

The staging frontend URL is responsible for the staging web boundary.

The Origami Software Engineer is responsible for the handoffs.

Trying to make one of those things own everything is where the system starts feeling cursed.

The better model is to let each boundary own exactly what it is supposed to own.

A staging release is not one deployment command.

It is a controlled handoff between environments, state, network boundaries, and immutable artifacts.

Once that becomes visible, the sequence stops looking annoyingly ceremonial.

It starts looking protective.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

Staging Frontend Deployment Using a Firebase Hosting Preview Channel

Aaroophan Varatharajan — Thu, 15 Oct 2026 07:46:01 +0000

A technical guide to staging frontend deployments with Firebase Hosting Preview Channels.

A technical guide to staging frontend deployments with Firebase Hosting Preview Channels, covering .firebaserc, firebase.json, build artifacts, channel URLs, authentication, deployment output, expiration, and backend frontend-origin configuration.

There is a very tempting way to think about staging frontend deployment.

The app is already hosted somewhere. There is already a live URL. There are previous releases. The hosting console has a neat deployment history. So the first instinct is simple:

“Can’t we just use one of those previous releases as staging?”

It sounds ✨reasonable✨

It is also where the mental model starts lying to us.

The live site is not staging

In Firebase Hosting, the live site has a job. It serves the current production version.

It is protective. Slightly boring. Very serious.

When the console shows domains like:

example.web.app
example.firebaseapp.com
custom-domain.com

Those are not staging URLs. Those are attached to the live Hosting channel.

The “Previous releases” section can also feel misleading at first. It looks like a version list. It has deploy hashes. It has timestamps. It has a calm little history of what happened.

But previous releases are not staging environments.

They are history.

That distinction matters because staging is not just “something that existed before.” Staging is a place where a new build can be viewed, tested, shared, and thrown away without touching the live channel.

The live channel says:

“I am what users see.”

A preview channel says:

“Come break things here first.”

That is the boundary.

The scary part: “deploy” does not always mean “production deploy”

This is the uncomfortable part.

The command contains the word deploy.

firebase hosting:channel:deploy stg --project <firebase-project-id>

And when production is nearby, the word “deploy” has villain energy.

It sounds like it might overwrite the live site. It sounds like it might replace the current release. It sounds like one wrong Enter key could turn a staging experiment into a cursed production moment.

But Firebase Hosting has separate channel concepts.

A normal Hosting deploy targets the live channel.

A preview channel deploy targets a named preview channel.

So this command:

firebase hosting:channel:deploy stg --project <firebase-project-id>

does not promote the build to the live site.

It creates or updates a preview channel called stg.

That distinction is the whole trick.

It is still a deployment, but it is not a live deployment.

Before deploying, check what Firebase thinks this project is

The next trap is assuming the CLI is looking at the right Firebase project.

That assumption is too confident.

Before running anything, check .firebaserc :

cat .firebaserc

A simple config may look like this:

{
  "projects": {
    "default": "<firebase-project-id>"
  }
}

That tells the Firebase CLI which project the local folder belongs to.

Then check firebase.json :

cat firebase.json

For a frontend app, it may look like this:

{
  "hosting": {
    "public": "dist",
    "ignore": [
      "firebase.json",
      "**/.*",
      " **/node_modules/**"
    ],
    "rewrites": [
      {
        "source": "**",
        "destination": "/index.html"
      }
    ]
  }
}

This tiny file is the gatekeeper.

It says: “When you deploy Hosting, I am not deploying your source code. I am deploying whatever is inside dist.”

That means the preview channel is only as correct as the build output inside dist.

If dist is stale, Firebase will faithfully deploy stale vibes.

Firebase is not wrong. It is obedient.

That is worse.

Build first, then deploy the preview

So the safe sequence is not to rush into the preview deploy.

First, confirm the branch:

git status
git branch

Then build the frontend:

npm install
npm run build

Then check that *dist * exists:

dir dist

or on Linux/macOS:

ls -la dist

Only after that does the preview channel command make sense:

firebase hosting:channel:deploy stg --project <firebase-project-id>

If a longer-lived preview URL is needed:

firebase hosting:channel:deploy stg --project <firebase-project-id> --expires 30d

The build step creates the artifact.

The preview deploy publishes the artifact.

Those are different responsibilities, and mixing them up is how staging starts gaslighting everyone.

The first failure is usually boring: authentication

A very normal first failure looks like this:

Error: Failed to authenticate, have you run firebase login?

This is not a Firebase Hosting problem.

It is the CLI saying:

“I don’t know who you are.”

The fix is straightforward:

firebase login

During login, Firebase may ask optional questions about Gemini features or CLI usage/error reporting. Those choices do not change the preview deployment itself.

After login succeeds, rerun:

firebase hosting:channel:deploy stg --project <firebase-project-id>

This is one of those failures that feels annoying because it is not intellectually interesting.

But it is useful.

It proves the CLI is not silently deploying with some mystery identity. It forces an authenticated boundary before touching Hosting.

That is a good kind of boring.

Reading the deploy output without panicking

A successful preview deploy may show output like this:

hosting:channel: Channel stg has been created on site <site-id>.

=== Deploying to '<project-id>'...

deploying hosting
hosting[<site-id>]: beginning deploy...
hosting[<site-id>]: found 8 files in dist
hosting[<site-id>]: file upload complete
hosting[<site-id>]: finalizing version...
hosting[<site-id>]: version finalized
hosting[<site-id>]: releasing new version...
hosting[<site-id>]: release complete

Deploy complete!

Hosting URL: https://<live-site>.web.app
hosting:channel: Channel URL (<site-id>): https://<preview-url>.web.app

This output can look suspicious at first because it still prints the live Hosting URL.

That does not mean the live site was replaced.

The important line is the channel URL:

Channel URL: https://<preview-url>.web.app

That is the staging frontend URL.

That is the URL to share.

That is the URL to put into the staging backend config if the backend needs to know the frontend origin.

The live URL appearing in the output is just Firebase being Firebase. Helpful, but slightly dramatic.

The preview URL becomes part of the staging system

Once the preview channel exists, the frontend has a temporary staging address.

That URL can then be wired into backend configuration:

FRONTEND_URL="https://<preview-url>.web.app"

This matters for flows where the backend needs to know the frontend origin, such as redirects, email links, CORS behavior, or environment-specific links.

The frontend preview channel is not only “a link to click”.

It becomes part of the staging environment’s identity.

That is where the preview channel stops being just a Firebase feature and starts becoming deployment architecture.

The frontend has its own temporary boundary.

The backend points at that boundary.

The staging environment begins to behave like an environment, not just a random build on the internet.

What this teaches

The seductive mistake is thinking staging is just another deployment.

It is not.

Staging is a boundary.

Firebase Hosting already has a live boundary. Preview channels give us a temporary one. The job is not to fight that model, or to reuse previous releases, or to squint at deployment history until it looks like staging.

The job is to respect the channel model:

Each step has ownership.

The branch owns the code.

The build owns the artifact.

firebase.json owns what gets deployed.

The preview channel owns the temporary URL.

The backend config owns which frontend it trusts.

The Origami Software Engineer owns the mental model.

Once that model is clear, the command stops feeling dangerous:

firebase hosting:channel:deploy stg --project <firebase-project-id>

It is not a risky shortcut around production.

It is the safe path around production.

And that is the broader lesson: good deployment workflows are not only about pushing code somewhere. They are about creating the right boundaries so code can be wrong in the correct place.

A staging frontend is allowed to be temporary.

It is allowed to be tested.

It is allowed to expire.

It is allowed to reveal that the mental model was incomplete.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

Checkout Didn’t Disappear. It Moved.

Aaroophan Varatharajan — Mon, 12 Oct 2026 04:04:27 +0000

How self-checkout, smart carts, checkout-free retail, accessibility, automation, and persuasive interfaces redistribute friction, responsibility, and control

A technical analysis of retail checkout automation covering POS systems, self-checkout, scan-and-go, smart carts, checkout-free stores, human-machine coordination, accessibility, failure recovery, employee workload, and persuasive interfaces.

Checkout begins with a wonderfully reasonable idea.

A shopper has already done the difficult part: walked the store, found the products, made the decisions. Now they just need to pay.

So make that final step ✨faster✨

The staffed POS terminal gives us a clean mental model. The cashier owns item registration, handles unusual products, resolves pricing problems, manages payment, and closes the transaction. The shopper mostly waits.

Then barcode mediation makes the cashier faster.

Self-checkout looks like the obvious next move.

Give the scanner to the customer.

Remove the queue.

Let the machine handle the boring parts.

This is seductive because it really does work, when the transaction is ordinary.

And that qualification is where the entire architecture starts to move.

Self-Checkout Works Until the Shopper Becomes the Operator

At self-checkout, the customer suddenly owns more of the sequence:

scan the item
place it correctly
interpret the prompt
choose the produce
confirm the quantity
pay
recover when something feels wrong.

The interface has not merely automated the cashier.

It has redistributed the cashier’s work.

That distinction is easy to miss because the machine looks independent while everything is going well.

Then a barcode refuses to scan.

Then the bagging state disagrees with the physical basket.

Then an age-restricted product appears.

Then payment stalls.

Now the cheerful kiosk becomes a tiny bureaucrat with one favourite sentence: Please wait for assistance.

Speed alone was never enough. Research on self-checkout quality links the experience to speed, ease of use, reliability, satisfaction and repatronage, not merely how quickly the scanner beeps. (IDEAS/RePEc)

The deeper problem is ownership.

The shopper has been given responsibility for the routine path without necessarily being given authority over the exceptional path.

That is why self-checkout can feel autonomous right up until the instant autonomy matters.

The Queue Moves Into the Aisle

The natural response is to move checkout earlier.

Why wait until the end?

Scan-and-go lets the customer register products while shopping. Smart carts push the same idea into the trolley itself, where the cart can maintain basket contents, running totals, product information, and sometimes route guidance or payment support.

Now checkout is no longer an event.

It is state.

That changes everything.

The system has to remember what has been scanned, what remains in the basket, what price applies, what discounts exist, and whether the shopper and the backend still agree about reality.

The smart cart becomes a very confident accountant rolling beside you.

Usually, that is useful.

Running totals externalise information the shopper would otherwise keep mentally. Smart-cart research also shows that usefulness and ease of use matter strongly to acceptance, while the cart remains only one part of the broader physical and social shopping experience. (EconBiz)

But the running total is not merely a nicer receipt delivered early.

It can change behaviour.

Budget shoppers may use feedback to spend closer to their intended limit. Non-budget shoppers may react to the accumulating total by reducing expenditure or substituting cheaper products.

The interface is no longer just reporting the basket.

It is participating in the basket.

Then We Try to Make Checkout Disappear

Once scanning has moved from the terminal to the aisle, the next temptation is almost irresistible.

Why make the shopper scan at all?

Let cameras, sensors, computer vision, account identity and backend inference reconstruct the transaction.

Enter.

Take products.

Leave.

Checkout-free.

On paper, this is the cleanest version yet.

But explicit scanning was doing two jobs.

It registered the product.

It also told the shopper, the system saw this.

Remove the scan and you remove friction, but you also remove a confirmation boundary.

Now the store may know something the shopper cannot immediately inspect.

Was the product registered?

Was it registered twice?

Which discount applied?

What happens if the virtual basket and the physical basket disagree?

The system has become smoother by becoming less visible.

And that is where trust stops being a vague branding concept and becomes an interaction requirement. Research on autonomous retail shows that higher perceived risk can reduce adoption, while retailer trustworthiness and consumer self-efficacy can mitigate that relationship. (IDEAS/RePEc)

The uncomfortable part is simple:

The more invisible the transaction becomes, the more important inspectability becomes.

The digital receipt, correction path, explanation, and dispute mechanism are not administrative leftovers.

They are part of the interface.

Failure Reveals the Real Architecture

Routine transactions hide architecture beautifully.

Failure does not.

An unreadable barcode, weight mismatch, duplicate item, payment interruption, audit, or uncertain recognition event suddenly forces the system to answer a harder question:

Who owns recovery?

The obvious solution is to give the shopper a retry button.

That works when the shopper caused something obvious.

But forcing the customer to diagnose a technological failure feels different, because responsibility has shifted without authority shifting with it.

Recovery therefore has to respect the apparent source of failure.

If the customer made a simple correctable mistake, guided self-recovery preserves autonomy.

If the technology failed — or nobody can tell what happened — the system or employee needs to take more responsibility.

And when control moves, the handoff needs state:

the current basket
the probable cause
actions already attempted
what the employee changed
what the customer should do next

Without that context, the employee is not receiving a transaction.

They are receiving a mystery.

That is why preserving completed work matters so much. Restarting the transaction may technically recover the system while absolutely torching the user experience.

The architecture is not just the checkout path.

It is the recovery path too.

Accessibility Exposes the “Default Shopper”

A touchscreen kiosk can look perfectly usable while quietly assuming an extremely specific human.

Standing.

Sighted.

Able to reach the scanner.

Comfortable with fine touch targets.

Able to hear or see the alert.

Able to understand the sequence quickly while other people wait.

Once those assumptions become visible, accessibility stops looking like an optional mode.

It becomes architecture.

Changing workstation geometry can improve posture and preference for wheelchair users without requiring the transaction to become slower. Alternative input can make touchscreen systems operable without relying on the touchscreen itself. Smartphone-mediated interaction can move control onto a device already configured for the user.

But even that apparently elegant answer has a boundary.

A phone assumes device ownership.

Battery.

Connectivity.

Account access.

Digital confidence.

We removed one dependency and quietly introduced four more.

So the requirement cannot be “put everything in the app.”

It has to be equivalent routes to the same transaction.

And those routes have to survive recovery too.

An accessible happy path followed by an inaccessible error dialog is accessibility with a trapdoor.

The Human Never Actually Left

The automation story becomes most revealing when we look behind the kiosks.

Self-checkout appears to reduce cashier work because customers scan and pay themselves.

But Moradi, Levy and Cheyre describe something more awkward:

Pseudo-automation.

Routine work moves to customers, while the remaining employee work becomes more parallel, more problem-oriented, and more focused on monitoring and policing. Traditional checkout can even become harder as simpler transactions migrate toward self-service. (alphaXiv)

The employee did not vanish.

They changed jobs while standing in roughly the same building.

One-to-one service becomes one-to-many supervision.

Scanning becomes exception handling.

Routine conversation becomes intervention.

And intervention often begins when the customer is already frustrated.

The worker becomes part technician, part monitor, part translator, part diplomat.

Sometimes they even perform relational patchwork : repairing the awkward social relationship created when the machine effectively says, something about this transaction looks suspicious.

This is why employee-facing interfaces matter.

If the customer sees “please wait for assistance” while the employee sees six identical red alerts, neither side has enough state.

Automation has created a coordination problem and then handed both humans a warning light.

A little cursed, when you put it that way.

The Checkout Starts Persuading You

Once the interface knows the basket, location, purchase history and loyalty state, another possibility appears.

Why only help the shopper complete the transaction?

Why not recommend something?

Show a personalised offer.

Display progress toward a voucher threshold.

Guide the route toward a promoted product.

Surface a complementary purchase.

This can be genuinely useful.

It can also become behavioural architecture.

The difficult line is not between persuasion and no persuasion.

Retail has always influenced choice.

The useful distinction is between three conditions.

Supportive influence helps the shopper pursue a goal they already chose.

Commercially aligned influence benefits both shopper and retailer.

Manipulative influence advances the retailer’s goal by obscuring alternatives, exploiting vulnerability, or making refusal unnecessarily difficult.

The mechanism may look almost identical on screen.

The difference is transparency, proportionality, control, and whose goal the interface is optimising.

That becomes especially important because data intensity and persuasive capacity rise together.

The more the system knows, the more precisely it can help.

The more the system knows, the more precisely it can steer.

Same infrastructure.

Different authority.

The Deeper Model: Friction Redistribution

At this point the original mental model no longer survives.

Checkout innovation is not a straight line from:

Human → Machine

Slow → Frictionless

What actually happens is friction redistribution.

Waiting moves.

Scanning moves.

Verification moves.

Error diagnosis moves.

Monitoring moves.

Privacy cost moves.

Employee workload moves.

The Origami Software Engineer moves.

Sometimes friction disappears from the shopper’s view because somebody else is carrying it.

That is why the most automated configuration is not automatically the most human-centred one.

The better model is interpretable hybridity.

Automation should reduce unnecessary effort while preserving:

understandable basket state
reversible correction
reachable assistance
accessible alternatives
coherent handoffs
inspectable automated decisions
fair responsibility.

And the evaluation has to happen at several levels at once.

The shopper cares about efficiency, comprehension, autonomy, accessibility and emotion.

The employee deals with workload, safety, discretion and relational demands.

The service system has to maintain accuracy, recovery, continuity and resilience.

Governance has to carry privacy, explainability, persuasion and contestability.

Optimise one carelessly and another may pay the bill.

That is the constraint that makes the final design feel inevitable.

Checkout did not need to become more human because automation failed.

It needed to become more interpretable because automation succeeded at distributing itself across everything.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The Distributed File System That Wins Depends on What You’re Afraid Of

Aaroophan Varatharajan — Fri, 09 Oct 2026 07:21:01 +0000

Why NFS, AFS, Coda, and GFS Make Different Trade-Offs for Different Failures

A technical comparison of Sun NFS, AFS, Coda, and GFS, examining how transparency, caching, scalability, availability, replication, consistency, reliability, and security constraints shaped each distributed file system.

At first, this looks like a comparison problem.

Put Sun NFS, AFS, Coda, and GFS in four columns.

Add transparency, performance, scalability, availability, reliability, security.

Give each system a score.

Highest total ✨wins✨

Perfectly reasonable.

And almost immediately wrong.

The problem is that these systems are not four contestants solving the same problem with different levels of competence. They entered the room carrying different fears.

NFS is worried about remote files feeling remote.

AFS is worried about thousands of clients exhausting the servers.

Coda is worried about the network simply disappearing.

GFS assumes machines will fail routinely and asks how to keep massive data processing moving anyway.

So the useful question is not: Which distributed file system is best?

It is: Best under which constraint?

Transparency Looks Easy Until the Network Starts Having Opinions

The first instinct is simple: if a file is remote, hide that fact.

Sun NFS takes this intuition seriously. Its protocol is explicitly designed for transparent remote access , with RPC carrying operations across the network and XDR providing a common representation across different machines, operating systems, network architectures, and transport protocols.

NFS is the diplomat here. It wants everybody to speak file system without caring too much about where the bytes live.

Its stateless server follows the same philosophy. The server avoids maintaining protocol state about its clients, which makes recovery after failure dramatically simpler: clients can retry operations instead of requiring the server to reconstruct an elaborate shared history.

That feels beautifully clean.

Then scale arrives.

AFS wants the same kind of location-transparent experience, but it expects thousands of workstations. If every read keeps dragging Vice, the trusted server side, back into the conversation, Vice eventually becomes the colleague who is somehow invited to every meeting.

So AFS gives Venus, the client cache manager, more responsibility.

Venus caches entire files locally. Reads and writes can then happen against the cached copy, while Vice keeps the responsibilities that actually require server authority. The system preserves a homogeneous, location-transparent namespace while deliberately pushing work toward clients.

The simple mental model was:

transparency means hiding the network.

AFS reveals the deeper model:

transparency also depends on deciding who must participate in each operation .

That distinction becomes surprisingly expensive later.

Mobility Exposes the Assumption AFS Was Still Making

AFS caching solves a lot.

But it quietly assumes that the server is still reachable when the client actually needs it.

That assumption feels harmless until a portable workstation leaves the network.

Now the cached copy is sitting there, perfectly usable, while the system technically has nobody authoritative to talk to.

The obvious conservative answer is to stop.

No server, no update.

Safe. Consistent. Miserable.

Coda refuses that bargain.

It inherits much of the AFS model but introduces two complementary mechanisms: server replication and disconnected operation. During disconnected operation, a caching site temporarily assumes the role of a replication site, allowing useful work to continue even when the shared repository is inaccessible. The design explicitly identifies portable workstations as a major use case.

This is where the architecture gets uncomfortable.

If a disconnected client can modify data, another client may modify its own copy elsewhere.

Now both copies are sincere.

Both have history.

Both think they are right.

Coda chooses an optimistic strategy because preventing all such divergence would destroy much of the availability it is trying to create. Conflict detection and later reconciliation become the price of continued operation.

That is the hidden constraint:

Mobility is not just remote access from another machine. It is maintaining useful authority when the normal authority is unreachable.

Once that requirement appears, “just stay consistent” stops being a complete answer.

Performance Stops Being a Number

It is tempting to ask which system is fastest.

Fast at what?

GFS makes this question impossible to ignore.

Its world contains large files, streaming reads, append-heavy writes, commodity machines, many clients, and applications that care more about high sustained bandwidth than low latency.

Under those assumptions, traditional file-system instincts start looking suspicious.

GFS uses a single master for metadata and many chunkservers for data. Clients ask the master where chunks live, then communicate directly with chunkservers for the actual data path. The master knows where everything is, but it very deliberately refuses to carry everybody’s luggage.

Large chunks reduce metadata overhead and repeated master interaction.

Replication assumes failures will happen.

Record append reflects the actual workload instead of pretending random overwrite is equally important.

This is specialization without apology.

AFS reaches performance from the opposite direction.

Its enemy is server interaction.

Whole-file transfer and client caching mean Venus can satisfy much of the workload locally. Callback-based cache validation further reduces the need for clients to repeatedly ask servers whether cached data is still valid. Howard and colleagues identify whole-file transfer and caching as central to AFS’s ability to scale gracefully.

So GFS says:

make the data path bypass central metadata control.

AFS says:

make repeated data access bypass the server entirely.

NFS stays more conservative. Its stateless RPC model gives it simplicity and portability, but that same simplicity keeps the client-server relationship comparatively chatty. Even the protocol specification acknowledges that server simplicity can become awkward when clients want richer file-system semantics.

Nobody is “faster.”

They are removing different bottlenecks.

Scalability Is Usually Someone Becoming Tired

Scale has a habit of revealing whichever component everyone politely assumed would cope.

In AFS, that component is Vice.

So Venus takes work away from it.

In GFS, that component could be the master.

So clients cache metadata and transfer data directly with chunkservers.

In NFS, statelessness keeps servers conceptually simple, but repeated remote interactions remain part of the model.

In Coda, scale is complicated further because availability mechanisms themselves create state: replicas must be tracked, cached versions must be reconciled, and disconnected clients eventually have to return carrying edits made in another reality.

This is why scalability is not just “add more machines.”

Adding machines adds:

more metadata,
more coordination,
more replicas,
more failure combinations,
more opportunities for stale state,
more people asking the same exhausted authority for answers.

The architecture scales when it decides which conversations no longer need to happen.

Availability and Reliability Look Similar Until Something Actually Fails

Coda and GFS both care deeply about failure, but their definitions of surviving it are different.

Coda asks: Can the user keep working?

Server replication protects against some server and network failures. Disconnected operation covers the much uglier case where no appropriate server can be reached at all.

GFS asks: Can the storage service keep functioning while components routinely die?

Its answer includes chunk replication, monitoring, master recovery mechanisms, stale-replica detection, and checksums. Google explicitly designed GFS around the assumption that component failures are normal in clusters built from inexpensive commodity hardware.

Same word: availability.

Different anxiety.

And reliability gets even more specific.

A replica existing somewhere is not enough.

What if it missed an update?

What if its disk corrupted the data?

What if a client reads from something that looks alive but belongs to yesterday?

GFS uses chunk version numbers to identify stale replicas and checksums to detect corrupted data. Replication is therefore not just duplication; replicas must remain trustworthy enough to participate. The verification work around GFS further examines its relaxed and eventual consistency behaviour, showing that consistency here is an explicit system property rather than a vague promise.

The replica does not get authority merely because it still has a pulse.

That is a useful distinction.

Data Integrity Reveals Why Availability Has a Price

The seductive model of replication is: make three copies and relax.

Distributed systems do not relax.

GFS coordinates mutations across replicas and uses versioning and checksumming to identify stale or damaged copies. Its relaxed consistency model is not equivalent to abandoning integrity; instead, the system defines what applications may observe and shapes the interface around workloads that can tolerate those guarantees.

Coda makes a different trade.

It wants updates to remain possible across disconnection and partitions.

That means temporary divergence can be legitimate.

The system therefore needs mechanisms for detecting conflicts and reconciling them later.

This can feel like integrity has been weakened.

A better way to see it is that the definition of integrity changed because the availability requirement changed.

GFS tries hard to keep replicas participating in one coordinated storage reality.

Coda sometimes permits two realities to exist temporarily because refusing the second reality would mean refusing the user.

Neither choice is free.

That is the part the tidy architecture diagrams usually leave out.

Security Is Where Old Trust Assumptions Start Showing

NFS began with portability, transparency, and simple recovery as dominant concerns.

Its classic design consequently places considerable trust in the surrounding authentication environment. The protocol’s early security model relies on mechanisms such as AUTH_UNIX, while the stateless server must repeatedly reason about permissions without maintaining rich client state.

That trade becomes increasingly uncomfortable as the environment gets less friendly.

AFS and Coda operate with a more explicit trusted-server model, and Coda’s design includes authentication and encrypted communication because mobility, caching, and replication make misplaced trust especially expensive.

GFS is different again. Its foundational design discussion spends enormous attention on failure tolerance, replication, throughput, and consistency, but adversarial multi-tenant security is not one of its central design axes.

That does not automatically mean “insecure.”

It means something more important when comparing architectures: absence of emphasis is not evidence of strength.

A system deserves credit for the problem it actually solves, not for requirements we wish it had discussed.

The Leaderboard Was the Wrong Abstraction

Once all these constraints are visible, the rankings become almost embarrassingly contextual.

Sun NFS is compelling when the problem is transparent remote access across heterogeneous systems and the architecture should stay comparatively simple. Its RPC/XDR model and statelessness follow directly from that goal.

AFS becomes compelling when the problem is large shared-workstation populations and servers cannot remain involved in every access. Whole-file caching, Venus, Vice, callbacks, and volumes are consequences of scale.

Coda becomes compelling when connectivity itself cannot be trusted. Server replication and disconnected operation follow from the requirement that users continue working through failures and mobility.

GFS becomes compelling when the environment is large, data-intensive, append-heavy, and built from hardware expected to fail. Large chunks, a metadata master, direct client-to-chunkserver communication, replication, checksums, and relaxed semantics all follow from that workload.

That is the deeper pattern.

Architecture starts looking strange when we stare only at the solution.

Once we reconstruct the constraint that came before it, the weird parts become almost inevitable.

The stateless server was not minimalism for vibes.

Whole-file caching was not an optimization somebody added because caches are fashionable.

Disconnected operation was not a feature bolted onto AFS.

GFS did not relax traditional semantics because correctness became unfashionable.

Each system moved because its original mental model eventually met a requirement it could no longer ignore.

That is what mature architecture means to The Origami Software Engineer:

Not a collection of universally correct choices,

but a record of which assumptions survived contact with the constraints.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

When "Just Make the Interface Easy" Stops Being Enough: Rethinking Human Computer Interaction

Aaroophan Varatharajan — Sun, 04 Oct 2026 14:18:39 +0000

Why goals, tasks, translation, ergonomics, interaction styles, error recovery, context, and perceived value shape usable systems

A technical exploration of Human-Computer Interaction covering Norman’s execution and evaluation gulfs, Abowd and Beale’s interaction framework, ergonomics, WIMP, interaction styles, error recovery, context, fluidity, and user value.

Interaction looks simple from a distance.

A user wants something. A system can do something. Put an interface between them, make the controls obvious, and the problem should mostly disappear.

That intuition is ✨reasonable✨

It is also where the trouble starts.

The deeper you follow interaction, the less it looks like a screen-design problem and the more it looks like two very different worlds trying to understand each other without exhausting, confusing, or misleading either side. (HCI Book)

The Seductive Model: User In, System Out

At first, interaction is simply communication between user and system.

That gives us a wonderfully clean picture: the user says something, the computer responds, and everybody goes home happy.

Except communication is not the same as understanding.

The interaction already carries four kinds of pressure: interaction models must translate between user and system, ergonomics must deal with physical characteristics, interaction styles determine the nature of the dialogue, and context brings social, organizational, and motivational forces into the room.

So the interface is not merely passing messages.

It is trying to keep two different realities synchronized.

Domain, Goal, and Task: The First Crack

The first useful separation is deceptively small.

A domain is the area of work under study.
A goal is what the user wants to achieve.
A task is how the user goes about achieving it through operations or actions.

Suppose the domain is graphic design and the goal is to create a solid red triangle. Selecting the fill tool and clicking the triangle is merely the task.

That distinction matters because interfaces are very good at making us obsess over tasks.

Buttons. Menus. Clicks. Steps.

But the user did not arrive because they wanted to click a fill tool.

They wanted the triangle.

Once that becomes clear, the interface stops being the purpose and starts becoming what it always was: the negotiator.

Norman Gives the Negotiator a Workflow

Donald Norman makes the interaction much more precise.

The user establishes a goal, formulates an intention, specifies actions at the interface, executes them, perceives the resulting system state, interprets that state, and evaluates it against the original goal.

This creates the execution/evaluation loop.

And it feels reassuringly orderly.

The user knows what they want. Their intention becomes an action. The system responds. The user observes the response and decides whether the goal was achieved.

This should work.

It does work.

Except when the user and system disagree about what any of those steps mean.

The Gulfs: Where a Correct System Can Still Feel Wrong

Now the hidden constraint becomes visible.

The Gulf of Execution appears when the actions the user formulates are not the actions the system allows.

The Gulf of Evaluation appears when the state the user expects is not how the system actually presents that state.

Nothing necessarily crashed.

Nothing necessarily malfunctioned.

The system may be completely correct internally while feeling almost hostile externally.

Then human error divides again.

A slip happens when the user understands the system and the goal, formulates the correct action, but performs the wrong action.

A mistake can begin earlier: the user may not even have the correct goal.

That distinction quietly changes responsibility.

A slip suggests better interface design.

A mistake suggests better understanding of the system.

"User error" was never one thing.

Abowd and Beale Reveal the Translation Problem

The next model makes the tension harder to ignore.

Abowd and Beale introduce four participants: User, Input, System, Output.

Each has its own language.

The user’s intentions must become actions at the interface. Those actions must become alterations of system state. That state must become output. The output must then become something the user can interpret.

Interaction is therefore translation between languages.

And the uncomfortable conclusion follows naturally:

Problems in interaction = Problems in translation.

The system is not necessarily broken.

The translation chain is.

That is why apparently small interface decisions can create surprisingly large misunderstandings. (HCI Book)

Ergonomics Reminds Software That Humans Have Bodies

It is tempting to keep solving this as a language problem.

Then ergonomics walks in carrying chairs, lighting, noise, physical position, button placement, colour, temperature, and human fatigue.

Suddenly the abstraction has a body.

Controls may need grouping according to function or frequency of use. Seating may need to accommodate different users. Colour choices need awareness of colour-blindness. The physical environment changes what is usable.

The difference between office and industrial interfaces makes this wonderfully obvious.

Office data may be textual, slow-changing, and used in a clean environment.

Industrial data may be numeric, fast-changing, and surrounded by dirt.

Then comes the perfect little reality check: The oil soaked mouse.

Nothing is technically wrong with the mouse.

The context simply made our "obvious" solution cursed.

Glass Interfaces Solve One Problem and Introduce Another

Replacing traditional dials and knobs with screens and keypads feels like progress.

A glass interface can be cheaper, more flexible, provide precise values, and show multiple representations of the same information.

Very seductive.

But physical controls knew where they lived.

The glass interface may lose physical location and context while becoming more complex.

So the solution is not always replacement.

Sometimes the design needs both.

That becomes even more important with indirect manipulation. In an office interface, the user often manipulates an artificial world directly. In an industrial interface, the user may manipulate the real world through the interface.

Now feedback and delays matter.

The interface may say the command was accepted while the physical process is still catching up.

The screen and reality are no longer the same clock.

Interaction Styles Are Different Kinds of Conversation

Once interaction is understood as dialogue, the next question becomes obvious: What kind of conversation should this be?

The command line interface gives direct access and works beautifully for repetitive tasks and expert users, but asks the user to remember its vocabulary.

Menus make options visible and replace some recall with recognition.

Natural language feels wonderfully familiar until vagueness and ambiguity arrive. The practical retreat is often to understand a subset or pick out key words.

Question/answer interfaces guide novices but restrict functionality.

Query languages such as SQL offer power, but demand understanding of structure and syntax.

Form-fills borrow familiarity from paper forms.

Spreadsheets extend that idea while maintaining consistency between values and formulas.

None of these styles is simply "better."

Each changes who carries the cognitive load.

WIMP Makes Interaction Easy, Then Becomes an Ecosystem

Then comes WIMP : Windows, Icons, Menus, Pointers.

Point-and-click interaction reduces typing. Windows divide the display into independent areas. Icons represent objects or actions. Pointers let the user select things. Menus expose available operations.

It feels almost effortless.

Then the family expands.

Buttons invoke actions. Radio buttons represent mutually exclusive choices. Check boxes support non-exclusive choices. Toolbars provide fast access to common actions. Palettes keep actions available when menus disappear. Dialogue boxes request information and then, ideally, go away.

Each component exists because another interaction created friction.

Menus save recall but consume screen space, so pop-up menus appear only when needed.

Toolbars provide speed but can become rows of mysterious icons whispering, "…but what do they do?"

Palettes exist because sometimes the menu is not there when you need it.

The interface keeps evolving because every simplification has a boundary. (HCI Book)

"Look" Was Never Enough

This is where interactivity starts to matter.

It is easy to focus on look.

The harder question is feel.

Two WIMP systems may contain windows, icons, menus, pointers, and buttons while behaving differently.

Hence: Appearance + Behaviour = Look and Feel

Speech-driven interfaces make the same lesson obvious in another form.

Speech recognition may misunderstand the user. A robust dialogue can therefore reflect its interpretation back:

"You want a ticket from New York to Boston?"

Now the user can correct it.

The system is not pretending to be perfect.

It is making misunderstanding repairable.

Initiative, Error, and the Right to Recover

Another hidden question appears: Who has the initiative?

Traditional question/answer interaction often gives it to the computer.

WIMP usually gives it to the user.

Then a modal dialog box appears and refuses to leave until the user responds.

Sometimes that is appropriate for errors or essential steps.

Use it everywhere and the interface becomes the colleague who keeps blocking the doorway to ask one more "quick question".

Errors themselves cannot always be avoided either.

So the design shifts from impossible perfection to error and repair.

Make the error easy to detect.

Then make repair possible.

That is a much more humane model of interaction because it accepts something the earlier mental model tried to hide:

Humans and systems will occasionally misunderstand each other.

Good interaction survives the misunderstanding.

Context Changes the Same Interface Without Touching the Code

By now it is tempting to believe we finally have the whole picture.

Then context widens it again.

Interaction is affected by other people, competition, desire to impress, fear of failure, allegiance, ambition, self-satisfaction, and organizational pressures.

An inadequate system can create frustration and reduce motivation.

The software may not have changed at all.

The interaction has.

That is the uncomfortable part: usability does not live entirely inside the interface.

The surrounding social and organizational environment can change how people behave with exactly the same system. (HCI Book)

"Can Use" Becomes "Want to Use"

For work systems, making interaction possible may sometimes be enough.

For home, entertainment, and shopping, another condition appears.

People must want to use it.

Now the conversation shifts toward experience, engagement and fun.

Experience can sit between anxiety and boredom.

Education may involve things a learner can just accomplish with help.

The interaction is no longer judged only by whether the task can technically be completed.

The user’s willingness to remain in the interaction becomes part of the design.

Physical Design Turns Certainty Into Trade-offs

Then physical design brings constraints that refuse to cooperate.

Ergonomic constraints say controls need sufficient size.

Physical constraints say there may not be enough room.

Safety constraints may favour one position for adults and another for children.

Aesthetic constraints want the design to look good.

Economic constraints quietly ask who is paying for all this.

There is no magic arrangement where every constraint wins.

There are design trade-offs.

The gas-cooker example makes this painfully clear: front controls may be safer and easier for adults, while rear controls may be safer for children.

The problem is not finding the "correct" position.

The problem is understanding which constraint each position satisfies.

Fluidity Is When Physical and Logical State Stop Arguing

That leads naturally to fluidity.

Do external physical aspects reflect logical effect?

Does the physical state reveal the logical state?

Do inverse actions create inverse effects?

Spring-back controls show why this matters.

They can support large selection sets efficiently, but they introduce hidden state because the control returns physically while the logical state may remain elsewhere.

The control looks innocent.

The state has moved on.

Now the user has to remember what the control no longer reveals.

A compliant interaction avoids that kind of drift where possible: state remains evident, and controls can reflect what both user and machine are doing.

Value Was the Constraint Hiding Under Everything

After all the models, controls, translations, ergonomics, styles, errors, and trade-offs, the final condition is almost embarrassingly simple.

People use something only if it has perceived value and that value exceeds cost.

Value might be getting work done, having fun, or doing something good for others.

Cost might be money, download time, or learning effort.

Then discounted future makes the equation nastier.

People heavily discount future value and future cost.

That helps explain resistance to learning: The learning effort is happening now, while the promised benefit lives somewhere in the future wearing a tiny "trust me" badge.

Hence the design pressure toward: Low barriers and high perceived present value.

Organizations face the same issue through coercion, enculturation, and emergence : force behaviour, build shared values, or design the process so individual value becomes organizational value.

Different mechanisms.

Same underlying problem.

The Interface Was Never the Real Goal

The general lesson sounds almost too simple after everything required to reach it:

If you want someone to do something, make it easy for them .

And understand their values .

"Easy" now means much more than fewer clicks.

The goal must survive translation into action.

The available actions must fit the user’s intention.

The system state must be perceivable and interpretable.

Errors must be repairable.

Physical controls must fit their environment.

Interaction styles must fit their users.

Constraints must be traded consciously.

The eventual value must justify the cost of learning and acting.

The Origami Software Engineer must keep all of this aligned.

That is why interaction design keeps growing beyond the interface.

Every time the model becomes more realistic, another hidden assumption becomes visible.

That’s not failure.

That’s evolution.

The "I liked this" Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

“Send” Is Not One Operation: Rethinking Distributed Computing #5

Aaroophan Varatharajan — Sat, 26 Sep 2026 13:24:58 +0000

The dangerous part of `send(message)` is how much meaning we quietly pack into the word “send.”

"Why send(message) is not one operation. Explore transient vs. persistent communication, synchronous vs. asynchronous messaging, queues, publish/subscribe, message brokers, MPI, collective communication, delivery guarantees, ordering, and the contracts hidden behind distributed communication."

Two processes need to communicate.

One has some data.

The other needs it.

So we do the obvious thing: send(message)

There. ✨Sent✨

Except distributed systems have a habit of taking perfectly ordinary verbs and asking uncomfortable follow-up questions.

Where is the message now?

Is it still in the sender’s memory?

Has the operating system accepted it?

Has it crossed the network?

Has the receiving machine stored it?

Has the receiving process seen it?

Has the receiving process processed it?

Can the sender disappear now?

Can the receiver be offline?

And when send() returns, which one of those things are we actually claiming happened?

That is the trap.

“Send” sounds like one operation because human communication lets us get away with vague semantics.

Distributed communication cannot.

The system eventually has to define exactly what survives, who waits, where the message lives, and what counts as success.

And once we start answering those questions, send(message) turns out to be several very different contracts wearing the same verb.

First Question: How Long Should the Message Stay Alive?

Suppose Alice sends a message to Bob.

Bob is not running.

What should happen?

One perfectly reasonable answer is:

Then the message cannot be delivered.

This is transient communication.

The communication system keeps the message only while the communicating applications and supporting communication infrastructure are active enough to handle it. If the receiver needs to be present when the message arrives, absence matters.

Transport-level communication often has this flavour.

The sender and receiver are participating in the same live conversation.

That works beautifully when both sides are expected to exist at roughly the same time.

Then consider email.

Alice sends a message at 2 p.m.

Bob opens his mail at 9 p.m.

Nobody finds this surprising.

We quietly introduced a completely different contract.

The communication system takes responsibility for preserving the message until Bob can receive it.

That is persistent communication.

The sender does not have to remain running after submitting the message.

The receiver does not need to be running when the sender submits it.

Somewhere between them, the infrastructure remembers.

The distinction is not really about messaging syntax.

It is about who owns the message while the receiver is absent.

Transient communication says:

I will carry this while the conversation exists.

Persistent communication says:

Give it to me. I will keep it until the other side can deal with it.

Same word.

Very different promise.

Second Question: When Is the Sender Allowed to Continue?

Now suppose Bob is available.

Alice sends the message.

Can Alice immediately continue doing other work?

That gives us another axis entirely.

With asynchronous communication, the sender submits the message for transmission and continues.

The message may still be sitting in a local buffer.

It may be inside an intermediate communication server.

It may not be anywhere near Bob yet.

Alice has handed off responsibility and moved on.

With synchronous communication, Alice waits.

But even this turns out not to have one obvious meaning.

What exactly is she waiting for?

Perhaps she waits until the message reaches a buffer on Bob’s machine.

That is receipt-based synchronization.

Or perhaps she waits until the message is actually delivered to Bob’s application.

That is stronger.

Or perhaps she waits until Bob processes the message and sends something back.

That is response-based synchronization.

Now the innocent verb send has become a ladder of guarantees.

Submitted.

Buffered.

Received by host.

Delivered to application.

Processed.

Responded to.

Each level tells Alice more.

Each level can also make Alice wait longer.

And this is the part that is easy to miss:

Persistent vs. Transient and Synchronous vs. Asynchronous are not the same question.

One asks:

How long can the communication system preserve the message?

The other asks:

How long must the sender wait?

Those two choices can be combined.

A message can be persistent and asynchronous, like the intuitive model of email.

Persistent and synchronous communication can wait until durable storage exists at the receiving side without requiring the receiving application itself to be running.

Transient asynchronous communication can let the sender continue immediately while relying only on temporary buffering.

Transient synchronous communication can wait for receipt, delivery, or even a response while still assuming the live communication context exists.

We did not make messaging complicated for fun.

We discovered that “sent” was hiding multiple independent decisions.

The Queue Says, “I’ll Hold It”

Once persistence matters, a useful character enters the story.

The message queue.

Instead of requiring Alice and Bob to meet directly, Alice puts a message into a queue.

Bob retrieves it later.

The queue sits between them like a patient intermediary saying:

You two do not actually need to coordinate your calendars.

Applications communicate by placing messages into queues and removing them when ready.

That produces loosely coupled communication.

Alice does not necessarily need Bob to be active.

Bob does not necessarily need Alice to still exist.

Neither side has to wait for the other to reach the exact same point in its lifecycle.

AWS describes SQS in essentially this role: a durable queue that helps decouple distributed components, allowing producers and consumers to operate without requiring them to be concurrently available.

That is a profound change in ownership.

In direct transient communication, Alice and Bob are responsible for meeting each other.

With persistent messaging, the queue accepts part of that responsibility.

And because it does, the interface itself changes.

Instead of one vague send, queue systems expose operations with much clearer meanings.

Put appends a message.

Get waits until a message exists and removes it.

Poll checks without waiting.

Notify arranges for a handler to be called when a message becomes available.

The verbs are more precise because the state of the communication is now explicit.

The message can exist independently of both applications.

Queued Does Not Mean Processed

This creates another seductive assumption.

If Alice successfully puts the message in the queue, perhaps the job is done.

Not quite.

What has actually happened?

The queue accepted responsibility for the message.

That is valuable.

It is not the same thing as Bob processing it.

Bob may not read the queue immediately.

Bob may fail while processing.

The message may remain waiting.

A queue can therefore give the sender confidence about handoff without automatically giving confidence about application completion.

Modern queue systems make this distinction visible. In Amazon SQS, for example, a message can remain stored until a consumer retrieves it; while being processed it can remain hidden for a visibility period, and the consumer deletes it only after successful processing. (docs.aws.amazon.com)

That lifecycle exists because these are different events:

The communication system has the message.

and:

The application has completed the work represented by the message.

Again, “sent” was too vague.

The queue forces us to become honest.

Then We Stop Naming the Receiver

Queues decouple sender and receiver in time.

But Alice may still be sending work toward a particular destination.

What if she does not know who should receive the information?

Suppose Alice produces an event:

Order 781 was completed.

Several systems may care.

Billing.

Analytics.

Inventory.

Notifications.

Perhaps systems we have not even built yet.

Making Alice explicitly call every consumer creates another dependency problem.

So we change the communication model again.

Alice becomes a publisher.

She publishes the event, usually to a topic.

Interested systems become subscribers.

The publish/subscribe infrastructure determines who should receive which messages.

Now communication has been decoupled in several dimensions.

The publisher and subscriber do not necessarily need to be online at the same time.

They do not necessarily need to know each other’s addresses.

And the publisher does not need to stop and wait for every subscriber to finish processing the event.

AWS’s SNS documentation describes this fan-out model directly: publishers send through a topic, which distributes messages to multiple subscribers; combined with queues, messages can also be persisted for later asynchronous processing.

The publisher says:

This happened.

Not:

Bob, please execute method X.

That is a different relationship entirely.

The components are becoming less aware of one another.

Which is excellent for loose coupling.

It also means the communication infrastructure is taking on more responsibility.

Somebody Still Has to Understand the Message

There is another hidden assumption.

Alice publishes:

{ customer: 17, total: 4500 }

Bob expects:

{ customerId: 17, amount: 4500 }

The message arrived perfectly.

Congratulations. Nobody understands each other.

Transport success does not imply semantic compatibility.

Applications still need agreed message formats, schemas, commands, and meanings.

One option is standardization.

Make every participant agree on the same format.

That works until integrating independently developed systems becomes painful.

Another option is a message broker.

The broker sits between applications and can transform incoming messages into representations understood by destination applications.

Now the intermediary is no longer merely storing and forwarding.

It may be translating.

Routing.

Filtering.

Mediating between different expectations.

The system has bought more decoupling by moving more responsibility into the communication layer.

There is always a bill somewhere.

A Message Bus Is Not a Conveyor Belt

Once many applications communicate through shared infrastructure, we can go further and define a message bus.

Applications agree on message schemas.

They agree on command messages.

They use shared infrastructure for sending those messages to recipients.

This feels pleasantly orderly.

Then comes another assumption worth losing:

If I sent A and then B, surely everyone receives A and then B.

Not necessarily.

Routing can change.

Messages can be buffered.

Internal optimization can alter paths.

Different transports may behave differently.

Some messaging infrastructures explicitly provide ordering guarantees, while others do not or provide them only under particular configurations. For example, AWS distinguishes standard queues, which can deliver messages more than once and occasionally out of order, from FIFO queues with stronger ordering guarantees. (docs.aws.amazon.com)

This is another reminder that ordering is a guarantee, not a vibe.

If application correctness depends on order, the communication contract must actually provide that order or the application has to reconstruct it.

By now, our simple message has acquired quite a résumé.

Persistence.

Synchronization.

Delivery guarantees.

Destination independence.

Schemas.

Routing.

Ordering.

And we still have not reached the strangest version of send.

MPI Wants Speed More Than Comfort

Message queues are excellent when components need loose coupling and persistent asynchronous communication.

But consider a very different environment.

A parallel scientific program runs across hundreds or thousands of processors connected by a high-speed network.

The processes are alive together.

They are cooperating on one computation.

Milliseconds of messaging middleware overhead may be deeply unwelcome.

Persistence after one process disappears is not necessarily the primary goal.

What matters is fast, explicit communication between parallel processes.

Enter the Message-Passing Interface, or MPI.

MPI is designed for distributed-memory parallel applications and provides standardized communication primitives that can be used across different hardware platforms.

It is deliberately more explicit.

Processes have ranks.

They belong to communicators.

Messages have destinations and tags.

The programmer controls how data moves.

This gives MPI performance and flexibility.

It also gives the programmer enough rope to construct some wonderfully sophisticated deadlocks.

The abstraction is no longer trying to pretend that communication is someone else’s problem.

MPI looks at the programmer and says:

You wanted control. Here is control.

`MPI_Send` Still Does Not Mean What You Think

Consider: MPI_Send(...)

It is a blocking send.

Easy.

So it must block until the receiver receives the message.

Right?

Not necessarily.

The MPI standard says that a blocking standard-mode send does not return until the send buffer can safely be reused. The implementation may accomplish that by copying the message into a temporary system buffer, or by transferring it into the matching receive. Whether buffering is used in standard mode is left to MPI.

That means MPI_Send can return even though the destination process has not yet executed its matching receive.

Now make the message larger.

The MPI implementation may decide that buffering it is too expensive.

The same call may now have to wait for the receiver.

The source code still says: MPI_Send(...)

The observed behaviour changed because the implementation’s buffering strategy changed.

This is why MPI programs that secretly depend on “small messages are buffered, therefore my send won’t wait” are living on borrowed time.

A different MPI implementation can expose the assumption.

A different buffer threshold can expose it.

A different message size can expose it.

The code did not become wrong because the message grew.

The code was already relying on a guarantee MPI never made.

That is an uncomfortable but useful distinction.

MPI Has More Than One Kind of Send for a Reason

Once we understand the ambiguity, MPI’s multiple send primitives stop looking like API clutter.

A buffered send explicitly uses buffering.

A synchronous send waits until the corresponding receive has begun.

A standard send leaves more of the buffering decision to the implementation.

Nonblocking operations allow communication to begin while the process continues with other work, with completion handled separately.

The MPI standard deliberately provides these communication modes because buffering changes both performance and synchronization behaviour; standard mode may buffer, while synchronous mode lets the programmer require stronger coordination with the matching receive. (mpi-forum.org)

These are not several spellings of the same operation.

They express different contracts.

And once again, the right question is not:

Which send function is coolest?

It is:

What must be true before this process is allowed to continue?

That is the recurring question underneath almost every communication abstraction we have touched.

Receiving Requires Its Own Contract

Sending is only half the story.

An MPI receiver must decide what messages it is willing to accept.

Messages can be matched using a source, a tag, and a communicator.

The communicator defines the group of processes participating in that communication universe.

The process rank identifies participants inside that group.

Tags distinguish messages that might otherwise look similar.

A receive may accept a message from a specific process or use MPI_ANY_SOURCE when the sender does not need to be known in advance.

And a blocking MPI_Recv waits until a matching message arrives.

This explicit matching exists because in a parallel program there may be many messages in flight at once.

The receiver cannot simply say:

Give me “the message”.

Which message?

From whom?

For which phase of the algorithm?

Inside which group of processes?

Communication correctness increasingly depends not just on moving bytes, but on ensuring the right communication operations find each other.

MPI has made the hidden relationship between sender and receiver visible.

Then Point-to-Point Becomes Awkward

Suppose several processes are independently calculating pieces of one result.

At the end, Process 0 needs the total.

We could build that using individual sends.

Process 1 sends its partial result to Process 0.

Process 2 sends its partial result.

Process 3 sends its partial result.

Process 0 receives everything and adds it.

That works.

Then the process count grows.

We can make the algorithm more clever by combining partial results through a tree.

Now processes exchange intermediate sums.

Fewer communication steps.

Better use of parallelism.

And eventually an obvious question appears:

Why should every program rebuild the same communication pattern manually?

That is where collective communication becomes natural.

Instead of describing dozens of individual point-to-point messages, express the group operation itself.

MPI_Reduce says:

Combine values contributed by the processes and place the result at one destination.

MPI_Allreduce says:

Combine them and give the result to everybody.

MPI_Bcast says:

One process has data; distribute it to the group.

MPI_Scatter says:

Divide data from one process among the group.

MPI_Gather says:

Collect distributed pieces onto one process.

MPI defines these explicitly as collective operations involving groups of processes rather than as mere convenience wrappers around an arbitrary collection of sends.

The abstraction has moved upward.

We are no longer saying how each pair of processes should communicate.

We are describing what the group is trying to accomplish.

A Collective Is a Group Contract

This is where collective communication can punish a point-to-point mental model.

Suppose Process 0 calls MPI_Reduce.

Process 1 decides to be clever and calls MPI_Recv instead.

They both seem related to receiving data.

They are not interchangeable.

The program is erroneous.

Collective operations have collective semantics.

The processes participating in the communicator need to participate compatibly.

The MPI standard requires all processes in the communicator’s group to call the collective routine, and collective operations must be executed in the same order across that communication group.

This can feel annoyingly strict until the deeper model becomes visible.

A collective operation is not:

Process 0 happens to receive several messages.

It is:

This group of processes is participating in one coordinated communication operation.

The communicator defines who belongs to that contract.

The operation defines what role their data plays.

The ordering lets MPI match collective operations correctly and select efficient implementation strategies.

Once we accept that, the rules stop looking arbitrary.

The operation is bigger than any one sender.

“Send” Was Really a Question About Responsibility

Look back at how far we travelled from: send(message)

Transient communication asked whether the receiver needed to be present.

Persistent communication moved responsibility for storing the message into the communication system.

Asynchronous communication allowed the sender to continue after submission.

Synchronous communication made the sender wait for stronger evidence.

Receipt-based synchronization waited for the receiving host.

Delivery-based synchronization waited for the receiving application.

Response-based synchronization waited for processing to come back around.

Queues separated sender and receiver in time.

Publish/subscribe separated them in space and knowledge of one another.

Message brokers took responsibility for translating between different message formats.

MPI moved toward explicit, high-performance communication where buffering and synchronization became programmer-visible concerns.

Collective communication went further and made the whole group operation the unit of meaning.

None of these mechanisms exists because engineers could not settle on one good send() function.

They exist because distributed applications need different answers to the same questions:

Who owns the message now?

How long will it survive?

Who must be alive?

Who has to wait?

What counts as delivery?

What ordering is guaranteed?

Who participates?

What happens when the operation involves an entire group instead of one sender and one receiver?

Once those questions differ, the communication operation has to differ too.

Make the Contract Say What the System Means

The broader mistake is treating communication like physical movement.

Alice had bytes.

Now Bob has bytes.

Done.

But useful distributed communication is rarely just about where the bytes are.

It is about what the participants are allowed to assume after the communication operation returns.

A queue lets the producer assume the infrastructure accepted responsibility.

A response-based call lets the sender assume processing reached a response.

A synchronous MPI send gives a different synchronization relationship than a buffered one.

A collective says something about an entire group, not merely two endpoints.

Those are semantic guarantees.

And stronger guarantees are not free.

Persistence requires storage.

Synchronization introduces waiting.

Loose coupling requires intermediaries.

Ordering restricts implementation freedom.

Collective coordination requires compatible participation.

Performance improves when we stop demanding guarantees we do not need.

Correctness improves when we stop assuming guarantees we were never given.

So the next time The Origami Software Engineer’s API says send,

the useful question is not simply:

Where will this message go?

It is:

What exactly will be true when this operation says it is finished?

That is when messaging stops looking like a collection of oddly named primitives and starts looking like a set of deliberate contracts between independent processes.

“Send” was never one operation.

We just started with a mental model too small to notice.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

A Remote Function Call Will Never Really Be Local: Rethinking Distributed Computing #4

Aaroophan Varatharajan — Sat, 26 Sep 2026 13:24:47 +0000

The API can hide the distance. It cannot erase what the distance changes.

"Explore the CAP theorem through practical examples of replication, network partitions, split brain, quorum, consistency, and availability and learn what distributed systems sacrifice when communication fails."

Suppose a client needs some work done by another component.

On one machine, this is boring in the best possible way.

Call a procedure.

Pass some arguments.

The procedure runs.

Get the result back.

✨Continue✨

Then we move that procedure onto another computer.

The first instinct is almost irresistible:

Why should the application care?

If the remote machine exposes the same operation, perhaps we can make calling it look exactly like calling a local procedure.

Instead of forcing every application developer to think in sockets, packets, addresses, message formats, and transport protocols, give them something familiar.

result = calculate(x, y)

The caller should not have to stare into the abyss and think:

open socket
encode x
encode y
construct message
send bytes
wait
receive bytes
decode result

That is the promise of Remote Procedure Call, or RPC.

Call a procedure on another machine just as you would call a local procedure.

Birrell and Nelson’s original RPC design was built around exactly this idea:

Provide communication between programs across a network through the familiar abstraction of a procedure call. (Microsoft)

It is an excellent abstraction.

Which is why it is worth understanding where the abstraction stops.

The Network Does Not Know What a Function Is

Before RPC can hide the network, something still has to use the network.

Distributed communication commonly happens over IP using transports such as TCP or UDP.

TCP gives us a connection-oriented byte stream.

UDP gives us connectionless datagrams.

Neither one understands: calculate(x, y)

The network understands messages and bytes.

So somewhere between the line of application code and the remote implementation, the procedure call has to be transformed into communication.

That is where RPC begins doing its little stage trick.

The application calls a client stub as though it were an ordinary procedure.

The stub builds a message and passes it to the communication middleware.

The message crosses the network.

On the server, middleware passes it to a server stub, or skeleton.

The server side reconstructs the arguments and calls the real local implementation.

The implementation returns a result.

Then the whole journey happens in reverse.

Server implementation → skeleton → middleware → network → client middleware → client stub → application.

The application sees: result = calculate(x, y)

The distributed system sees a round trip between two independent processes.

The ONC RPC specification describes essentially this model: the caller sends a call message containing the procedure parameters, waits for a reply message containing the results, and resumes once that reply arrives. (RFC Editor)

The procedure call did not cross the network.

A representation of the procedure call did.

That distinction is going to become expensive.

Even the Arguments Cannot Travel as Themselves

Suppose x is an integer.

Easy enough.

Put the integer into the message.

Except two machines may not even represent that integer identically.

One architecture may use one byte order.

Another may use another.

Strings may have different encoding or alignment rules.

Floating-point representations have to agree.

Arrays and structures have to be flattened into something that can travel and reconstructed at the other end.

Memory addresses are even more suspicious. A pointer that means something inside the client’s address space does not suddenly become meaningful inside the server’s address space.

So the stub has another responsibility.

Marshalling.

Data is encoded into a transmissible representation.

On the receiving side it is unmarshalled, or decoded, back into usable data structures.

Protocols therefore need agreed representations for parameters and results. ONC RPC, for example, uses External Data Representation to describe the messages exchanged between participants. (RFC Editor)

This is the first crack in the illusion.

A local call can hand another procedure values that already live in the same computational world.

A remote call has to translate one machine’s world into something another machine can reconstruct.

The API may look local.

The data most definitely knows it travelled.

Then We Have to Find the Procedure

There is another detail a local call gets almost for free.

When code calls a local procedure, the runtime knows where that procedure is.

A remote service may be somewhere else entirely.

Which machine?

Which process?

Which endpoint?

Which version?

So remote invocation needs binding.

A client has to discover and connect to the service that implements the desired interface.

That may involve some kind of directory or registry.

Once bound, the client can continue pretending that it simply possesses something callable.

This is location transparency doing useful work.

The client wants:

Give me the service that performs this operation.

It does not necessarily want:

Please make me personally manage the physical location and network configuration of the process currently implementing it.

Even formal RPC specifications separate the call protocol from the higher-level mechanism that binds a client to a particular service and transport endpoint. (RFC Editor)

Again, perfectly reasonable abstraction.

Again, the machinery underneath has not disappeared.

We have hidden where the call goes.

We still have to deal with what happens while it is going there.

Local Calls Have a Very Comfortable Timeline

Consider an ordinary synchronous procedure call.

You call.

You wait.

The procedure returns.

You continue.

RPC can reproduce that programming model.

But once the operation becomes communication, other interaction patterns suddenly become useful.

The client may send a message and wait for a response.

Or send without waiting at all: fire and forget.

Or send and wait only for an acknowledgement that the request was received.

A client may block.

It may poll.

It may provide a callback.

The call may be asynchronous or use deferred synchronization.

The ONC RPC model itself notes that implementations are not restricted to the simple blocking model; asynchronous execution is possible so the client can continue doing useful work while the remote operation proceeds. (RFC Editor)

Why does a supposedly local-looking call suddenly need all these options?

Because latency exists.

A local function call is usually close enough that waiting is the natural default.

A remote invocation can spend meaningful time crossing a network, waiting in another process, doing work, and crossing the network again.

Distribution changed the cost of waiting.

So even before anything fails, the network has already leaked into the programming model.

Then something fails.

The Timeout Is Where the Illusion Gets Cursed

The client sends a request.

Nothing comes back.

So it waits.

Eventually it times out.

What happened?

That sounds like a simple question. It is not.

Perhaps the request packet was lost before reaching the server.

Perhaps the request reached the server, but the acknowledgement was lost.

Perhaps the server executed the procedure and the reply was lost.

Perhaps the network became unavailable.

Perhaps the server process failed.

Perhaps the entire server machine failed.

From the client’s point of view, several very different realities can collapse into exactly the same observation:

No reply arrived.

This does not happen in the same way with an ordinary local procedure call.

The network has introduced ambiguity.

And ambiguity is where “remote is just like local” finally stops being an innocent simplification.

Fine. Retransmit It.

For ordinary communication failures, there are reasonable techniques.

Give requests unique identifiers or sequence numbers.

Use acknowledgements.

Detect duplicates.

If a message appears to have been lost, retransmit it.

This works beautifully for a lost packet.

Client sends request.

No acknowledgement.

Timeout.

Client sends request again.

Server receives it.

Reply arrives.

_Problem solved.

_Except there is another possible history.

Client sends request.

Server receives it.

Server performs the operation.

The acknowledgement or reply disappears.

Client times out.

Client sends the request again.

Now the server has seen the same operation twice.

For something harmless, perhaps that does not matter.

For an operation with side effects, it matters quite a lot.

Imagine the remote procedure is: TransferMoney(100)

The client did not ask: MaybeTransferMoneySomeNumberOfTimes(100)

Retries make the transport more reliable.

They can simultaneously make the operation less obviously correct.

This is why duplicate detection exists. A server can recognize an identifier it has already processed and avoid executing the same logical request again. ONC RPC similarly uses transaction IDs to match requests and replies and describes retaining those IDs as a way of obtaining a degree of execute-at-most-once behaviour. (RFC Editor)

The retry mechanism is not merely recovering lost packets.

It is trying to recover the meaning of the original invocation.

Did the Server Run It?

Now we reach the unpleasant question.

The client sends a request.

The server executes it.

Before the client receives the reply, communication disappears.

What should the client conclude?

The tempting answer is:

The call failed.

But what exactly does failed mean?

It certainly means the client did not receive a successful result.

It does not necessarily mean the server did not perform the operation.

That is the hidden difference.

The client knows what it observed.

It does not automatically know what happened on the other machine.

Even with a reliable transport such as TCP, receiving a reply allows the caller to know the remote operation executed, but receiving no reply does not prove that it never executed. The RPC specification states this distinction explicitly. (RFC Editor)

That gives a timeout a very different meaning from the one our local-call intuition wants to assign to it.

A timeout does not say:

The procedure did not execute.

It says:

I can no longer determine the outcome from the communication I received.

That sounds like a small wording difference.

It changes application design.

Retries.

Idempotency.

Error handling.

Transaction semantics.

Recovery.

All of them eventually run into the same question:

Did nothing happen, or did something happen and I simply failed to hear about it?

The network can make those realities indistinguishable from the caller’s side.

Failure Transparency Has a Limit

RPC originally aimed for a highly transparent programming model.

Hide location.

Hide communication.

Hide parameter conversion.

Make remote interaction feel ordinary.

But failures are where complete transparency becomes dangerous.

If a local-looking call can fail because another machine disappeared, the caller eventually needs some way to know that this failure category exists.

Modern RPC designs therefore acknowledge the things the abstraction cannot safely erase: network failures, remote process failures, communication latency, asynchronous behaviour, and the possibility that a request’s outcome is uncertain.

A remote call is still a convenient abstraction.

It is just no longer pretending that distribution has no semantic consequences.

This is a healthier contract.

The abstraction says:

You do not need to manually construct network messages for every invocation.

It does not say:

You may reason as though there is no network.

Those are very different promises.

Then We Tried It With Objects

Once procedures could be invoked remotely, the next progression was natural.

What about objects?

That leads from RPC toward Remote Method Invocation, or RMI.

Instead of invoking a standalone remote procedure, a client invokes a method on a remote object.

The client first binds to that distributed object.

A proxy representing the remote interface lives on the client side, playing a role similar to the RPC client stub.

The proxy serializes or marshals the invocation into messages.

On the server, a skeleton receives the invocation, unmarshals it, and calls the actual object’s implementation.

Results travel back in the opposite direction.

But an important boundary remains.

The remote object’s entire memory does not magically teleport into the client.

What becomes remotely accessible is its interface.

The interface says what operations can be invoked.

The implementation and state still live elsewhere.

Java RMI makes that separation explicit: a remote interface declares the methods that can be invoked from another virtual machine, and failures during remote invocation include communication problems as well as marshalling and protocol errors. (Oracle Docs)

The object may look near.

Its state is still remote.

Remote Exceptions Are the Abstraction Admitting the Truth

RMI produces one particularly honest design choice.

Remote exceptions.

Suppose the remote method itself throws an application exception.

That failure can be marshalled into a response, sent back to the client, reconstructed, and raised as an exception on the caller’s side.

From the application’s point of view, that can feel pleasantly local.

But remote invocation also has failure modes that exist because the method is remote.

The server may be unreachable.

The connection may fail.

Arguments may fail to marshal.

The response may fail to unmarshal.

The protocol may fail.

Java RMI therefore requires remote interfaces to account for RemoteException, which represents communication-related failures that can occur during a remote invocation. (Oracle Docs)

That requirement is almost philosophical.

The API still gives us the elegance of: RemoteObject.DoSomething()

But the type system quietly taps us on the shoulder:

Remember. This can fail in ways a local method cannot.

That is not the abstraction giving up.

That is the abstraction becoming honest.

Transparency Should Remove Work, Not Reality

RPC succeeded because the original idea was good.

The Origami Software Engineer should not have to rebuild networking primitives every time one program needs to ask another program to do something.

Stubs are useful.

Marshalling is useful.

Binding is useful.

Location transparency is useful.

Remote interfaces are useful.

Turning messages into something that resembles ordinary procedure or method invocation is one of the most productive abstractions in distributed computing. The original RPC work and later standardized RPC protocols both preserve that familiar call model while explicitly dealing with binding, message representation, transport, and failure semantics. (Microsoft)

The mistake is not using the abstraction.

The mistake is believing it literally.

A local procedure call and a remote procedure call may share syntax.

They do not share physics.

One crosses an address-space boundary.

The other crosses machines, networks, representations, clocks, failure domains, and potentially administrative boundaries.

One can usually tell you whether control returned.

The other may leave you asking whether the operation happened at all.

So good distributed abstractions do something subtler than making the network disappear.

They hide the mechanics that programmers should not have to repeat.

Then they expose the consequences that programmers cannot safely ignore.

The remote function call was never going to become truly local.

It did something more useful.

It became familiar without pretending that distance was free.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

Navigating the CAP Theorem: Rethinking Distributed Computing #3

Aaroophan Varatharajan — Sat, 26 Sep 2026 13:24:29 +0000

Replication gives us another copy of the truth. Then the network disappears, and both copies have to decide whether they are still allowed to call themselves true.

"A practical guide to the CAP theorem, exploring consistency, availability, network partitions, split brain, quorum, and why distributed systems must choose what correctness means when communication fails."

Imagine a hotel booking system running on one server.

Bob asks for Room 12.

The server checks.

Room 12 is available.

Bob gets the room.

There is something ✨beautifully✨ uncomplicated about this. One server owns the state, so there is one obvious answer to the question:

Is Room 12 available?

But one server is also one failure away from having no booking system at all.

So we do the sensible thing.

Add another server.

Perhaps one serves one region and another serves somewhere else. Keep their booking records synchronized. If one machine becomes unavailable, the other can continue serving customers.

More redundancy.

More availability.

More confidence.

Until Bob asks Server A for Room 12 at exactly the wrong moment, while Dong asks Server B for the same room.

Normally, the servers coordinate.

This time, the connection between them has disappeared.

Server A checks what it knows.

Available.

Server B checks what it knows.

Also available.

Bob gets Room 12.

Dong gets Room 12.

Both servers behaved perfectly according to the reality they could see.

And together they produced an impossible hotel.

That is the uncomfortable beginning of CAP.

The Problem Is Not That Either Server Is Wrong

At first, it is tempting to blame synchronization.

Surely the servers should simply synchronize before confirming the booking.

That does solve the problem while communication is working.

But now keep the network partition.

Server A cannot reach Server B.

What should A do when Bob asks for the room?

There are two obvious choices.

It can say:

I cannot verify the current state, so I refuse to complete the request.

The data remains safe from contradictory bookings.

But Bob has encountered an unavailable service even though Server A itself is alive and perfectly capable of computing.

Or Server A can say:

I cannot reach B, but according to my local state the room is free, so I will continue.

Bob gets a response.

The service remains available.

But Server B may independently tell Dong exactly the same thing.

The system cannot guarantee both behaviours simultaneously.

That tension is what the CAP theorem formalizes.

In the usual formulation:

Consistency means clients observe one coherent, up-to-date view of the data.
Availability means requests to non-failing parts of the system receive responses.
Partition tolerance means the system must cope with communication being lost between parts of the network.

Gilbert and Lynch formally showed that in an asynchronous network subject to partitions, a distributed service cannot guarantee both strong consistency and availability during such a partition.

The network has forced a decision.

Not because the architecture was badly implemented.

Because the two servers no longer have enough information to make both promises.

“Pick Two” Is a Little Too Comfortable

CAP is often introduced as a triangle:

Consistency. Availability. Partition Tolerance.

Pick two.

It is memorable.

It is also dangerously easy to interpret as a shopping list.

Want consistency and availability? Choose CA.

Want availability and partition tolerance? Choose AP.

Want consistency and partition tolerance? Choose CP.

The diagram makes these choices look wonderfully symmetrical.

Reality is less cooperative.

In an actual distributed system, we do not control whether a cable is cut, a switch fails, packets disappear, or two healthy machines temporarily lose their ability to communicate.

We do not get to configure: network.partition = falseand move on with our day.

AWS’s current CAP guidance makes the practical implication explicit: distributed systems generally have to tolerate network failures, so when a partition occurs, the meaningful trade-off becomes consistency versus availability.

That changes the question.

It is no longer:

Which two letters would you like?

It becomes:

When communication breaks, what is the system allowed to sacrifice?

That is a much less decorative question.

And a much more useful one.

CP Says “I Would Rather Refuse Than Lie”

Go back to Room 12.

Server A cannot communicate with Server B.

If we care strongly about consistency, Server A cannot confidently approve Bob’s reservation.

Perhaps B has already sold the room.

A does not know.

That uncertainty matters.

So a CP-style design may refuse operations that cannot be completed safely while the partition exists.

This can feel frustrating.

Both servers are alive.

The database process is running.

The CPU is fine.

The request arrived.

And the system still says no.

But the refusal is doing real work.

The system is protecting an invariant:

One room must not be successfully reserved twice.

Consistency becomes more important than allowing every surviving partition to continue independently.

This same idea appears in distributed locking and majority-based protocols. A node may be technically alive yet deliberately prevented from acting because it lacks enough agreement from the rest of the system.

The service has not forgotten how to process the request.

It has lost the authority to process it safely.

That distinction matters.

AP Says “I Would Rather Answer and Reconcile Later”

Now imagine a different kind of data.

You edit a file while disconnected from the network.

Should the application freeze until every other copy of that file is reachable?

That would certainly make synchronization easier.

It would also be a fairly cursed user experience.

So an AP-style system can make the opposite trade.

Let the reachable side continue operating.

Accept that different replicas may temporarily disagree.

Then reconcile them later.

This leads naturally toward weaker consistency guarantees, including eventual consistency.

The distinction is useful:

Strong consistency says that after an update completes, subsequent accesses should observe that updated state.

Weak consistency does not guarantee that an immediate subsequent access will see it.

Eventual consistency adds an important promise: if updates stop, replicas will eventually converge toward the latest state.

Dropbox gives a wonderfully tangible version of the idea. Files marked available offline can be opened and edited without network access. Those changes do not synchronize to other devices while the machine is disconnected; synchronization resumes when connectivity returns. If the same file is changed independently, conflicts may need to be represented and resolved.

The local copy is allowed to keep living its life.

For a while, two devices may disagree.

That does not automatically mean the system has failed.

It means the system decided that temporary inconsistency was preferable to making the file unusable whenever synchronization became impossible.

AP Is Not “No Consistency”, and CP Is Not “No Availability”

This is where the simple CAP triangle can really mess with the mental model.

An AP system does not necessarily throw consistency into a bin and wish everyone luck.

A CP system does not necessarily collapse into permanent unavailability whenever one packet goes missing.

These properties have degrees, time windows, scope, and application-specific meaning.

An available system might return slightly stale information while aggressively propagating updates and resolving conflicts.

A consistency-focused system might remain completely available whenever communication is healthy and only restrict a subset of operations during an actual partition.

The trade-off becomes interesting precisely because it is not always static.

Consider an airline reservation system.

Suppose a flight has 200 seats and only 40 have been sold.

A temporary inconsistency in the exact remaining count may be tolerable. Refusing every reservation merely because one replica cannot be reached could cause more harm than allowing another booking.

Availability has room to breathe.

Now imagine 199 seats are gone.

Suddenly the remaining state matters a lot.

Two independent servers both believing that one seat remains can sell seat 200 twice.

The invariant has become tight.

Consistency now deserves more authority.

Same system.

Same data.

Different moment.

Different acceptable trade-off.

The deeper lesson is not “choose AP” or “choose CP.”

It is to understand which guarantees matter for which operations, and under which conditions.

Then the Backup Decides the Primary Is Dead

CAP becomes even easier to feel when we stop talking about database reads and watch a system accidentally create two authorities.

Consider a primary server and a backup.

Clients normally use the primary.

The primary sends updates to the backup so the backup can take over if necessary.

This is sensible redundancy.

Now break the communication link between primary and backup.

The backup stops hearing from the primary.

What should it conclude?

The primary might have crashed.

Or the network between them might simply be broken.

From the backup’s side, those situations can look painfully similar.

Suppose the failover rule says:

If the primary disappears, become the new primary.

The backup promotes itself.

Excellent.

Except the original primary never crashed.

It is still alive.

Some clients can still reach it.

Other clients can reach the newly promoted backup.

The old primary thinks:

The backup is unreachable. I am still primary.

The backup thinks:

The primary is dead. I am now primary.

Both conclusions are locally reasonable.

Globally, they are disastrous.

The system has developed a split brain.

Oracle’s clustering documentation describes the same failure pattern: after communication between cluster nodes is lost, separate partitions can each believe they are the valid cluster and continue operating independently.

Our redundancy has produced two sources of authority.

Both Halves Can Be Correct and the Whole Can Still Be Wrong

Split brain is nasty because neither side needs to be malfunctioning internally.

Imagine clients on the left continuing to write state through the original primary.

Clients on the right write different state through the promoted backup.

For a while, each half can look healthy.

Requests arrive.

Responses go out.

Data changes.

Nobody is necessarily throwing errors.

That is what makes the condition dangerous.

The problem is not that the machines stopped working.

The problem is that agreement about who is allowed to speak for the system disappeared.

Suppose the service answers a question whose correctness depends on information from both sides.

Each half can respond based on incomplete state.

Each response may be perfectly sensible from the local view.

Together, those answers can violate the assumptions of the system above them.

Split brain is CAP wearing a name tag.

The partition created separate realities.

Availability allowed those realities to continue making progress.

Consistency became impossible unless one reality stopped making independent decisions.

And now we can finally see what the solution really needs.

Not better optimism.

Not another timeout.

Agreement.

Two Machines Cannot Resolve the Argument by Staring at Each Other

The obvious fix sounds easy:

Let the primary and backup decide which one is really alive.

Unfortunately, that is almost the problem we already had.

If they can communicate reliably enough to make that decision together, they are not experiencing the partition we are trying to survive.

During the partition:

Primary cannot see backup.

Backup cannot see primary.

Each one knows only:

I am alive, and the other side is not answering.

That does not reveal whether the other side is dead.

Separate one-to-one decisions can therefore produce separate answers.

So the system needs something beyond the two conflicting participants.

One option is brutally simple:

Use one server.

If it fails, wait.

Consistency becomes straightforward, but availability during the failure suffers.

Another option is fencing: when a backup takes over, ensure the old primary is physically or logically prevented from continuing to serve.

If only one side is capable of acting, there cannot be two primaries.

And another option is majority agreement.

Now the partition must prove that it represents enough of the system before it is allowed to continue.

This is where quorum enters the story.

Majority Does Something Clever With Authority

Suppose there are enough voting members that one side of a partition contains 60 votes and another contains 40.

Only the side with more than half is allowed to continue as the authoritative cluster.

The other side may be alive.

It may have healthy disks.

It may have clients begging for service.

It still stops.

Why?

Because two disjoint groups cannot both possess a strict majority of the same electorate.

That mathematical detail is doing the heavy lifting.

It prevents two partitions from simultaneously claiming legitimate authority.

Oracle Solaris Cluster uses this exact idea: a partition with a majority of votes obtains quorum and is allowed to operate, while partitions without quorum cannot continue as the cluster. The mechanism exists specifically to prevent split-brain operation.

Notice what we paid for that guarantee.

The minority side became unavailable.

That is not an accidental flaw in the quorum mechanism.

That is the trade-off.

We deliberately sacrifice some availability because allowing both sides to continue would risk divergent state.

The architecture stops looking pessimistic once the constraint is visible.

A majority vote is not there because distributed-systems engineers enjoy making healthy servers shut themselves down.

It is there because being alive is not the same thing as being authoritative.

Partitions Do Not Break Reality. They Break Shared Reality.

This is the part of CAP that feels more useful than memorizing the triangle.

Before the partition, replicas can coordinate enough to present a common state.

Then communication disappears.

Each side continues observing its own local reality.

From that moment, the system cannot magically know what is happening beyond the partition boundary.

If both sides continue accepting conflicting updates, availability survives but immediate consistency cannot.

If the system refuses operations until sufficient agreement exists, consistency survives but some requests cannot proceed.

If users are allowed to work offline, their state may diverge temporarily and reconcile later.

If two servers both decide they are primary, redundancy becomes split brain.

If quorum grants authority to only one partition, the system stays coherent by deliberately silencing the other.

Same underlying problem.

Different manifestations.

The network did not merely disconnect machines.

It disconnected knowledge.

And once knowledge is separated, perfect consistency and perfect availability cannot both be promised as though nothing happened.

The Choice Was Never Really About Three Letters

CAP is memorable because of C, A, and P.

But the useful engineering lesson sits underneath them.

Distributed systems create copies of state because copies are useful.

They improve scalability.

They improve resilience.

They put computation and data closer to where they are needed.

But every copy introduces another participant that must somehow remain aligned with the others.

As long as communication works, synchronization can hide much of that complexity.

A partition removes that luxury.

Now the design has to reveal what it really values.

Can stale state be tolerated?

Can conflicting updates be reconciled later?

Is there an invariant that must never be violated?

Can an operation safely stop?

Who has authority during disagreement?

How is that authority established?

What happens when communication returns?

Those questions are far more useful than asking whether a database has an AP or CP sticker on the box.

The hotel booking system did not fail because we added a second server.

The backup did not become dangerous because redundancy was foolish.

The offline file was not broken because two copies temporarily disagreed.

Each design encountered the same underlying fact:

Independent machines cannot maintain one shared reality without communication, and communication itself is one of the things a distributed system cannot guarantee.

So the architecture adapts.

Sometimes it waits.

Sometimes it refuses.

Sometimes it keeps going with stale state.

Sometimes it reconciles.

Sometimes it asks a majority who is still allowed to speak.

Sometimes it checks with The Origami Software Engineer.

The network split reality in two.

The system’s job was never to pretend that did not happen.

It was to decide what correctness means when it does.

That’s not failure.
That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

— Aaroophan Varatharajan

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The Hard Part Was Finding the File: Rethinking Distributed Computing #2

Aaroophan Varatharajan — Fri, 25 Sep 2026 19:47:03 +0000

Decentralizing the files was easy. Decentralizing the answer to “where is the file?” was where things got interesting.

"How peer-to-peer systems evolved from Napster’s centralized directory and Gnutella’s query flooding to structured overlays, DHTs, and Chord’s logarithmic routing."

Peer-to-peer file sharing begins with an idea that feels ✨simple✨

Alice has a file.

Bob wants the file.

Why place a giant server in the middle?

Let Bob download it directly from Alice.

Better yet, let every participant behave as both client and server. A peer can download resources from others while contributing its own resources back to the system. The machines sitting at the edges of the Internet stop being passive consumers and start becoming part of the infrastructure.

That gives peer-to-peer systems much of their appeal.

There is no need for one server to store every file. Storage grows as peers join. Network resources are contributed by the participants themselves. The system can potentially grow far beyond what one central machine could comfortably serve.

So far, so good.

Then Bob asks the question we quietly skipped:

Where is Alice?

More precisely:

Which peer currently has the file I want?

And suddenly file transfer is no longer the interesting problem.

Lookup is.

Four Innocent Questions

A peer-to-peer system has to answer four basic questions:

Join: How do I begin participating?

_**_Publish:** How do I advertise what I have?

_**_Search:** How do I find what I want?

_**_Fetch:** How do I retrieve it?

Fetch is almost comforting.

Once Alice knows that Bob has the file and knows how to contact him, they can establish a direct connection and transfer it.

The awkward work happens before that.

Somewhere, somehow, the system must connect:

“The thing I want”

to:

“The peer that currently has it”

This is harder than it looks because peers are not dependable little servers sitting obediently in a data center.

They join.

They leave.

Their addresses can change.

They may disappear without warning.

Millions of resources may be spread across machines that know only a tiny fraction of the rest of the network.

So we need a lookup mechanism.

The first solution is exactly the one most of us would probably build.

Put the answers somewhere.

Napster Keeps the Map

Napster’s architecture made a clever distinction.

The files remained on users’ machines, but a centralized directory maintained information about which peers had which files. When a peer connected, it informed the central service about its address and available content. A search went to that directory; the resulting file transfer happened directly between peers. Stanford’s description of the original design similarly notes that the central servers stored information about shared files rather than the MP3 files themselves.

This is a beautiful compromise.

The expensive data is distributed.

The knowledge about the data is centralized.

Alice asks the directory:

Where is file A?

The directory replies:

Bob has it.

Alice talks directly to Bob.

Lookup is wonderfully cheap from Alice’s point of view. The directory already knows the answer. Search scope is effectively constant rather than requiring Alice to wander around interrogating peers.

And because the music files themselves are not moving through the directory server, it can concentrate on indexing and discovery.

This works.

It works really well.

Which is why the trap matters.

The directory has quietly become the one machine carrying the system’s collective knowledge.

Every peer tells it what is available.

Every search depends on it.

As the system grows, it maintains more state and handles more processing.

And if that directory becomes unavailable, the files may still exist on thousands of machines while users lose the mechanism for finding them.

That is a deeply cursed kind of availability:

The data is everywhere. The map to the data is gone.

Napster decentralized storage without decentralizing discovery.

The central directory was simple, efficient, and controllable.

It was also a performance bottleneck and a single point of failure.

So the next instinct seems obvious.

Remove it.

Gnutella Fires the Librarian

Suppose we refuse to maintain a central directory.

Now no machine knows where everything is.

Fine.

Alice can ask her neighbours.

If one of them has the file, excellent.

If not, they can ask their neighbours.

And those neighbours can ask theirs.

This is query flooding.

Gnutella embraced this decentralized model. A peer joins by finding existing peers and establishing connections. To search, it sends a query to its neighbours. They forward it onward, and matching peers return results. The Gnutella protocol describes every participant as both client and server, with queries propagated through the peer network rather than resolved by one central search server.

We solved Napster’s uncomfortable dependency.

There is no central directory.

There is no single server performing every lookup.

Search processing is spread across the network.

One peer disappears?

The rest can continue.

That feels properly peer-to-peer now.

Until Alice searches for something.

Her query goes to several neighbours.

They forward it.

Those peers forward it.

More peers see it.

More messages appear.

The system has traded:

One machine knowing too much

for:

Many machines being asked too much.

The lookup scope can approach O(N).

And peer-to-peer networks are exactly the kind of networks where N is supposed to become enormous.

Flooding is democratic.

It is also extremely enthusiastic.

Decentralization Did Not Make Search Free

This is where the intuition starts to break.

Removing centralized control certainly removes a central bottleneck, but it does not remove the work that the central service was doing.

That work has to go somewhere.

Gnutella distributes it.

Every query recruits part of the network into finding an answer.

That creates an ugly relationship between growth and search traffic. More peers give us more resources, but also a larger search space. Historical discussion of Gnutella’s design likewise noted that searching required queries to be forwarded across neighbouring nodes and became increasingly cumbersome as the network grew.

We can limit the depth of flooding.

We can use random walks rather than shouting in every direction.

Those reduce the blast radius.

But something deeper is still missing.

The network has very little structure connecting a resource to its location.

If Alice asks for file A, there is no inherent reason that the query should travel toward one particular part of the network.

The network does not know where to look.

It only knows whom it can ask.

That difference turns out to be enormous.

Maybe All Peers Shouldn’t Be Equal

There is another uncomfortable observation.

Peer-to-peer systems like to say peer as if every participant were equivalent.

Reality is less polite.

One peer may have a fast connection and remain online for days.

Another may have limited bandwidth and disappear five minutes later.

Treating them identically has egalitarian vibes, but not necessarily good systems design.

So systems such as KaZaA moved toward hierarchical query flooding using supernodes.

Ordinary peers connect to a supernode and publish information about their files there. A search first goes to the local supernode. If the answer is elsewhere, supernodes communicate with other supernodes.

This is interesting because we have partially reintroduced what we just removed.

Not one global central directory.

But not perfect flat decentralization either.

A hierarchy appears.

Some peers take on greater responsibility because some peers are simply better positioned to do so.

Search now happens among a smaller set of capable nodes instead of blindly involving everybody.

We have learned something slightly annoying:

Removing hierarchy is not the same thing as removing bottlenecks.

Sometimes hierarchy is useful.

The question is where to put it and how much of it to trust.

We Keep Trying to Search Better

Other peer-to-peer designs push the problem in different directions.

BitTorrent, for example, makes swarming the star of the show. Once the participants in a swarm are known, different pieces can be retrieved from different peers rather than treating one peer as the complete source.

Unstructured overlay systems such as Freenet take another route. Instead of simply flooding every query, peers use information gathered from previous requests to make better routing decisions.

Now the network has started learning.

A request is not necessarily broadcast blindly. It can be forwarded toward a peer that seems more promising.

That is progress.

But the routing remains unstructured.

The knowledge is heuristic rather than deriving from a global relationship between a resource identifier and a particular place in the overlay.

A query can become smarter without becoming predictable.

And predictability matters when the system is supposed to operate at global scale.

We would really like to say something stronger than:

“Hopefully the search goes somewhere useful.”

We want bounds.

We want guarantees.

We want a way to know that adding another million peers does not turn every lookup into a million-message group chat.

So the problem changes again.

Maybe we should stop asking:

How do we search the network?

and ask:

Can we organize the network so the resource itself tells us where to route?

That is the pivot.

Stop Searching. Start Routing.

A Distributed Hash Table, or DHT, changes the abstraction.

Think of an ordinary hash table.

Give it a key.

Get a value.

A DHT tries to provide a similar abstraction while the table itself is spread across many independent machines.

Conceptually:

put(key, value)

get(key)

The clever part is that keys and peers are placed into an identifier space according to defined rules.

Now a resource does not merely have a name that somebody must search.

It has an identifier that determines which part of the distributed system is responsible for it.

The lookup problem has become a routing problem.

That changes everything.

Instead of asking arbitrary peers whether they happen to know where a resource lives, the overlay can progressively route a request toward the peer associated with the key.

This is structured overlay routing.

Systems such as Pastry assign identifiers to nodes and objects and use routing tables to move requests toward progressively closer identifiers. The structure lets lookup complete in a bounded number of routing hops rather than relying on network-wide flooding.

Structure costs something.

Peers must maintain routing information.

Membership changes must be handled.

The mapping between objects and nodes has to remain usable while machines join and leave.

But now the cost buys us something flooding never really gave us: Predictability.

Chord Puts Everyone on a Ring

Chord gives this idea an unusually clean shape.

A hash function is used to produce identifiers for both keys and nodes. Those identifiers occupy the same circular identifier space.

Think of the numbers arranged around a ring.

A key is assigned to its successor: The first node whose identifier is equal to or follows the key’s identifier around that ring.

So if the desired object hashes to key **60**, the system does not ask every peer:

Does anyone have 60?

The identifier space already tells us which node should be responsible for 60.

Chord’s fundamental service is deliberately narrow: given a key, map that key to a node. Its authors describe efficient data location as the core P2P problem, and Chord solves it by providing this distributed key-to-node mapping while adapting as nodes join and leave.

That sounds like the problem is solved.

Almost.

Because knowing which identifier should own the key is not the same as knowing how to contact that node efficiently.

And this is where Chord becomes a lovely little exercise in engineering trade-offs.

Knowing Everyone Is Too Expensive

Suppose every Chord node keeps the address of every other node.

Lookup becomes easy.

Hash the key.

Find its responsible node in the local routing table.

Contact that node directly.

One network hop.

Lovely.

Except every peer now maintains O(N) routing state.

We have recreated a global-knowledge problem inside every machine.

Fine.

Go to the opposite extreme.

Each node knows only its immediate successor on the ring.

Routing state becomes tiny.

But now finding a distant key means walking successor by successor around the ring.

Lookup can require O(N) messages.

Wonderful.

We have two solutions:

Know everybody and route quickly;
Know almost nobody and route slowly.

Neither feels like the scalable answer we wanted.

The interesting question is whether there is something between everything and almost nothing.

There is.

The Finger Table Knows Just Enough

Chord gives each node a finger table.

Instead of remembering every other peer, a node keeps references at exponentially increasing distances around the identifier ring.

Roughly speaking, it knows someone nearby.

Then someone farther away.

Then much farther away.

Then farther again.

The routing table is not trying to know the whole world.

It is trying to know enough strategically positioned parts of the world to make progress quickly.

Suppose the desired key is far away.

The current peer chooses the finger that moves the request closest toward that key without overshooting the useful region.

The next peer does the same.

Then the next.

Each hop can remove a large portion of the remaining identifier-space distance.

It has the feeling of binary search, except the search is being performed through messages between independent computers.

Now the trade-off becomes much nicer:

O(log N) routing state per node.

O(log N) messages per lookup.

The Chord analysis shows exactly this logarithmic scaling of both communication cost and per-node state.

That is the moment the architecture stops looking clever for cleverness’ sake.

Finger tables exist because the two obvious alternatives forced us there.

Global knowledge was too expensive.

Successor-only knowledge was too slow.

So each peer remembers just enough structure to cut the search space down quickly.

The solution feels earned.

The Network Finally Knows Where to Look

Look back at the journey.

Napster said:

Ask the machine that knows everything.

Fast.

Simple.

Also centralized.

Gnutella said:

If nobody knows everything, ask everybody you can reach.

Decentralized.

Resilient to losing one directory.

Also expensive.

Hierarchical approaches said:

Maybe ask the better-connected peers first.

More practical.

But still based heavily on distributed search.

Unstructured routing said:

Use what the network has learned to make better guesses.

Smarter.

But without strong lookup bounds.

Then DHTs changed the shape of the problem:

Give resources and peers structure, then route toward responsibility.

Chord pushes that idea into a ring and gives every node just enough routing state to move efficiently through it.

We did not discover one universally perfect peer-to-peer architecture.

We kept changing the organization because each previous organization exposed a different cost.

Centralized knowledge created a bottleneck.

Removing the center created flooding.

Treating all peers equally ignored heterogeneity.

Unstructured routing lacked strong guarantees.

Maintaining complete routing knowledge would not scale.

Maintaining almost none would make lookup crawl.

The design kept moving because the lookup problem kept refusing to disappear.

Structure Is Not the Enemy of Decentralization

There is a broader lesson hiding in all of this.

It is easy to associate decentralization with the absence of structure.

No central server.

No hierarchy.

No authority.

Everyone equal.

Ask anyone.

Route anywhere.

That sounds beautifully distributed.

The Origami Software Engineer noted,

it also turns out to be a poor guarantee that the system will scale.

Peer-to-peer systems still need organization.

They need rules for joining, publishing, searching, and fetching.

They need to cope with machines constantly appearing and disappearing.

They need to balance load.

They need to avoid assuming every node has equal capacity.

And once they become large, they need lookup mechanisms whose cost does not grow disastrously with the population.

The useful distinction is not: _Centralized vs Structured.

_It is: Centralized Authority vs Distributed Structure.

Chord has structure everywhere.

The identifier space is structured.

Resource placement is structured.

Routing state is structured.

Lookup is structured.

Yet no single directory needs to know the entire network.

That is the evolution worth remembering.

We began by putting the map on one server.

Then we threw the map away and shouted.

Then we built partial maps.

Eventually, we stopped treating discovery as an uncontrolled search and designed the network so that the destination could be approached systematically.

The files were never the hardest part.

Knowing where to look was.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The One-Computer Illusion: Rethinking Distributed Computing

Aaroophan Varatharajan — Fri, 25 Sep 2026 19:32:36 +0000

Why making many independent computers behave like one coherent system is where the trouble actually begins.

"Explore how independent machines, networks, concurrency, partial failures, clocks, transparency, and coordination turn “just add more computers” into a fundamentally different engineering problem"

Suppose a system needs more processing power, more storage, better availability, or simply needs to serve more people.

The obvious instinct is: ✨Add machines✨

That sounds almost embarrassingly straightforward. One computer is doing too much work, so give it friends. Divide the workload. Share the resources. Keep going as demand grows.

And there are plenty of reasons to do exactly that. Some problems are inherently distributed. Some become too large for a single machine. Systems need to scale, remain available when components fail, share resources, serve geographically distributed users, and evolve without replacing everything at once.

So we add another computer.

Then another.

Then a hundred.

The trap is subtle: Somewhere during that progression, it is very easy to keep reasoning as though we still have one computer, just bigger.

We do not.

A parallel system may have many processors while remaining tightly coupled around shared resources. A distributed system gives us something fundamentally less comfortable: Independent Computers.

Each has its own processor.

Its own memory.

Its own execution.

Its own problems.

And if one machine needs something held by another, it cannot simply reach into the other machine’s memory.

It has to ask.

Through a message.

Across a network.

That tiny change is where the vibes start getting cursed.

So We Make Them Pretend to Be One

There is a wonderfully ambitious definition of a distributed system: A collection of autonomous or independent computers that appears to its users as a single coherent system.

Read both halves carefully.

Independent computers.

Single coherent system.

Those ideas are almost fighting each other.

The machines are autonomous, yet we want the user to experience coherence. They may run different hardware, operating systems, software, and protocols, yet the interaction should feel consistent. They communicate by passing messages, yet we want applications to behave as though the machinery underneath is one dependable whole.

That gives us one of the central design goals: Single System View.

The user should not have to care which machine stores an object, which server answers a request, which replica is currently active, or which node disappeared five seconds ago.

Ideally, the system handles that mess.

The user sees one thing.

The distributed system quietly negotiates reality underneath.

Beautiful.

Except the machines have not stopped being independent just because we hid them behind a nice interface.

The Network Gets a Vote

On one machine, communication between components can rely on mechanisms built around shared local resources.

Across independent computers, coordination happens through message passing. Coulouris’ characterization makes this constraint explicit, and with it come three consequences that keep resurfacing throughout distributed computing: concurrency, independent failures, and the lack of a global clock.

Now consider what our innocent abstraction has inherited.

The network might be slow.

A message might be delayed.

Communication has latency.

Bandwidth is not infinite.

Topology can change.

Different machines can be running different technologies.

And the network itself can fail.

These facts sound painfully obvious when written down. Yet many distributed designs become fragile precisely because they quietly assume the opposite:

The network is reliable.

Latency is zero.

Bandwidth is infinite.

Transport is free.

The network is secure.

Topology does not change.

There is one administrator.

The infrastructure is homogeneous.

Each assumption makes the system feel a little more like one computer.

Each assumption is also something reality is free to revoke.

The abstraction is not necessarily lying to us.

But it is definitely leaving out some terms and conditions.

There Is No Single “Now”

The machines are also doing things concurrently.

Process A may execute while Process B executes somewhere else. They progress independently, exchanging messages only when coordination requires it.

This sounds manageable until we ask a very ordinary question:

What is the current state of the system?

On a single machine, that question already has nuances.

Across independent computers, it becomes much nastier.

There is typically no process holding perfectly current knowledge of everything happening everywhere. While one node learns that something changed, another node may already have changed again. Messages carrying that information take time to arrive.

There is no convenient global memory containing “the state”.

There is also no perfectly shared global clock quietly settling every argument about which event happened first.

Each computer has a local clock, and clocks do not advance in perfect agreement. Interaction models therefore have to reason explicitly about timing, communication delays, and the difference between synchronous and asynchronous assumptions.

So the system we wanted to behave like one computer cannot casually rely on two things a one-computer mental model makes very seductive:

One authoritative global state

**and

**One authoritative global time

This is where the illusion starts demanding real engineering.

Even Failure Stops Being Simple

Then something breaks.

On a centralized system, a catastrophic machine failure is at least brutally clear: The machine is down.

Distributed systems introduce a stranger condition.

Partial failure.

One component can fail while the rest of the system continues running.

A process crashes.

Another process is fine.

A network link disappears.

Two healthy machines become unable to communicate.

A message is sent but never arrives.

And, wonderfully, the surviving processes may not immediately know what happened.

That last part matters.

A remote machine that has crashed can look suspiciously similar to a remote machine that is simply taking a long time to respond.

The system is no longer choosing between: Working or Broken

It can be Working and Broken at the same time.

That is why distributed systems need explicit failure models: omission failures, communication failures, arbitrary failures, timing failures, and mechanisms for detecting, masking, tolerating, or recovering from them.

Fault tolerance exists because adding machines did not merely add resources.

It added independent failure boundaries.

Fine. Hide All of It.

At this point the natural reaction is almost defensive.

If distribution is causing all this complexity, hide the distribution.

That is transparency.

Make local and remote resources look alike through access transparency.

Let users access resources without knowing their physical location through location transparency.

Hide replication so multiple copies appear to be one resource.

Allow resources to move without breaking applications.

Hide concurrency where possible.

Mask failures so a failed component does not automatically become the user’s problem.

Even allow the system to scale or reconfigure without forcing applications to change.

This is how those independent computers begin to recover the illusion of one coherent machine.

And transparency is incredibly useful.

But there is a trap inside the solution.

More transparency is not automatically better.

Trying to hide every aspect of distribution can hurt performance or conceal behavior that applications genuinely need to understand. Research and classic treatments of distributed systems likewise describe distribution transparency as a major goal while recognizing that locality and distributed-system complexity cannot simply be wished away.

A remote operation still crosses a network even if its API looks local.

A retry still costs time even if failure recovery is invisible.

A replicated resource still needs coordination even if users see only one logical object.

Transparency can hide complexity from the interface.

It cannot repeal the mechanisms underneath it.

The Illusion Gets Surprisingly Expensive

Now the original plan looks different.

We added machines for scalability, but centralized services, centralized data, and centralized algorithms can become bottlenecks as the system grows.

We wanted availability, but partial failures mean the system must detect, mask, tolerate, and recover from faults.

We wanted one Single System View, but concurrency and message passing mean nobody automatically owns the current global state.

We wanted transparency, but excessive transparency can fight performance.

We wanted an extensible system, but growth introduces heterogeneity across hardware, software, middleware, protocols, locations, and administrators.

This does not mean distribution was a mistake.

It means our first mental model was incomplete.

The machines were never pieces of one physical computer.

They were autonomous participants being persuaded to cooperate.

Model the System We Actually Have

Once that is accepted, the design starts looking less mysterious.

Instead of reasoning vaguely about “the system”, we can model different parts of the problem explicitly.

A physical model asks what computers and devices exist and how they are interconnected.

An architectural model asks which entities communicate, how they communicate, what roles they have, and where they are placed. Client-server and peer-to-peer structures become architectural choices rather than incidental diagrams.

A fundamental model goes deeper into assumptions that affect almost every design: How processes interact, How timing behaves, How components can fail, and What security threats exist. Coulouris’ framework explicitly uses architectural and fundamental models to reason about properties such as performance, reliability, scalability, interaction, and failure.

This is the useful shift.

We stop asking:

How do we make these machines behave exactly like one computer?

And start asking:

Which differences must we hide, which must we manage, and which must the design acknowledge?

Now message passing makes sense.

Fault tolerance makes sense.

Replication makes sense.

Caching makes sense.

Different architectural roles make sense.

Even the awkward edge cases make sense.

They are not random complications bolted onto an otherwise simple system.

They are consequences of asking autonomous computers to cooperate while presenting something coherent above them.

One System, Carefully Negotiated

The magic of a distributed system is not that many computers somehow become one computer.

They never do.

They remain independent machines, executing concurrently, communicating through messages, observing only pieces of global reality, running on imperfect clocks, and sometimes failing without permission from the rest of the system.

The achievement is that, despite all of that, The Origami Software Engineer can design enough communication, coordination, replication, fault tolerance, transparency, and structure that a user can often interact with the whole thing as though it were one coherent system.

That distinction matters.

Because once we stop believing the illusion literally, distributed-system design becomes easier to reason about.

The network is not an implementation detail.

Failure is not exceptional.

Global knowledge is not free.

Transparency is not magic.

And coherence is not something multiple machines naturally possess.

It is something the system has to continuously create.

We started by adding computers because one machine was no longer enough.

Then we discovered that every new independent machine also added another piece of reality that had to be coordinated.

That’s not failure.

That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan

The Fastest Way to Make Software Slow: Rethinking Rapid Application Development

Aaroophan Varatharajan — Fri, 25 Sep 2026 19:14:32 +0000

What Rapid Application Development taught me about speed, trade-offs, classic mistakes, and getting troubled projects back under control.

Rapid Application Development isn’t just about coding faster. Learn how effective practices, trade-offs, risk, scope, and project recovery create real speed.

I used to think speed was something you could ✨force✨ into software.

Pick the fastest tool. Use the newest framework. Tighten the deadline. Get good developers into a room, add enough coffee, keep everyone focused, and move.

The Product Manager encouraged this fantasy. At the beginning, he wanted quality, controlled feature creep, a predictable ship date, usability, performance, maintainability, testing… the whole respectable engineering starter pack.

Then Schedule walked into the room.

Usability? We don’t have time.

Performance? It can wait.

Maintainability? Next project.
Testing? Users want it now.

Just get it out the door.

That instinct is so seductive because it sounds practical. But rapid development means something narrower than “doing everything well, only faster”. It means speedy development, shorter schedules, development speed. Individual tools can help, but none of them applies to every case, they have to be orchestrated as part of a full-fledged strategy. The same body of work is explicitly organized around rapid-development strategy, classic mistakes, risk management, scheduling, teamwork and project recovery.

And that was the first thing I had to unlearn.

Fast is a priority, not a magical property that makes every other priority free.

Find the One Fast Thing

Everyone had a favorite shortcut.

The Hacker wanted to code for 36 hours at a stretch. The Information Engineer trusted CASE tools, intensive user involvement, and tight time-boxes. The programmer trusted rapid prototyping. The Management had whatever shiny practice had recently acquired good vibes.

Technology, naturally, loved this arrangement.

Technology showed up wearing a cape and whispered, I can save all of you.

It was lying to me.

Not because the tools were bad. They were often useful. The mistake was treating one useful thing as the whole strategy.

The real choice happens twice:

Choose effective practices rather than ineffective practices;
Among those, choose practices oriented specifically toward the schedule objectives.

That sounds almost disappointingly ordinary.

I wanted a trick.

What I got was a filtering problem.

And then Schedule split itself into 3 personalities:

Speed-oriented practices wanted actual development to move faster.
Risk-oriented practices wanted to prevent schedule overruns.
Visibility-oriented practices wanted progress to become visible.

Sometimes the problem wasn’t speed at all.

Sometimes we were moving fine but taking too much risk.

Sometimes the software was progressing but the customer couldn’t see it.

I had been treating “faster” as one problem.

It was already 3.

Speed Finally Gets Adult Supervision

The Four Pillars

Eventually four rather unglamorous characters arrived and ruined the shortcut party:

Classic mistake avoidance just carried a notebook containing every terrible idea the industry had already tried.
Development fundamentals was the older engineer in the corner asking whether we had actually done the engineering.
Risk management kept asking cursed questions about what would happen if assumptions failed.
Schedule-oriented practices were the exciting one, obviously. They promised movement.

They were the four pillars.

I wanted to skip the first three and hang everything from Schedule.

That is approximately how you turn architecture into modern art.

Rapid development is not a quick fix for something already collapsing. The foundation has to support the speed. The goal first becomes efficient development, a sensible balance of schedule, cost and product characteristics, before deliberately tilting toward a better schedule.

That distinction saved me from another fantasy: maximum schedule, minimum cost and maximum product quality at the same time.

You can optimize one direction harder.

You cannot pretend the trade-offs stopped existing because the deadline looked important.

People, Process, Product and Technology Enter the Room

Then the problem became 4-dimensional.

People had the greatest leverage, which was inconvenient because People refused to behave like a compiler flag. People had ability, motivation, team spirit, fatigue and opinions.
Process was easier to measure but had its own personality. Good Process removed friction. Bad Process built a ceremonial maze and then seemed offended when nobody moved quickly.
Product turned out to be surprisingly negotiable. Every feature Product carried made Schedule carry something too.
Technology was useful when chosen properly and deeply cursed when selected because somebody wanted to learn it during production.

The interesting part was synergy. Get the four dimensions working together and their combined effect can exceed their isolated contributions.

But first Process needed rehabilitation.

I had assumed Process was bureaucracy wearing a tie.

Instead, its best argument was rework avoidance.

Requirements that change late can demand redesign, recoding and retesting. Design problems discovered during system testing can throw completed work back into the furnace. One of the simplest ways to save time is therefore brutally obvious:

Don’t do the same work twice.

Quality Assurance joined Process with an equally unpopular message:

Catch errors near the moment they are introduced, because defects get more expensive and time-consuming the longer they survive.

Quality Assurance wasn’t slowing us down.

It was trying to stop Future Us from paying interest on Present Us’s laziness. (Us’s? Us’? Someone correct me in the comments.)

Product Learns the 80/20 Rule

Product had been quietly expanding while everyone argued about velocity.

One more feature.

One more requirement.

One more “small” thing.

Then Product explained the 80/20 rule:

A large portion of useful functionality may consume a relatively small portion of development time, while a smaller set of difficult features can consume most of it.

That changed the conversation.

If Product could reduce the feature set, Schedule could shrink without asking People to become nocturnal mammals.

If Product could keep characteristics flexible, Technology could use pre-existing components rather than custom code.

If we could distinguish needs from wants, some scope could be negotiated rather than heroically implemented.

The relationship between product size and effort wasn’t comfortably linear either. Smaller pieces are easier to design, build and test, reducing size can buy disproportionate schedule gains.

Suddenly the fastest code was sometimes the code we chose not to write.

Technology liked that too. Reuse orientation, components, frameworks, APIs and higher-level tools could remove work instead of merely making hands move faster.

But Product had one warning.

If we insisted simultaneously on exceptional performance, memory use, robustness, reliability, usability, feature richness and schedule, somebody eventually had to pay.

Usually People.

Usually at night.

The Shortcut That Sends You Backward

There is another road.

Hire strong people. Demand total commitment. Give them autonomy. Motivate them aggressively. Work 60, 80, perhaps 100 hours a week.

Code like hell.

I understand why this path keeps getting chosen.

For a while, it feels incredible.

People move fast. Messages arrive at ridiculous hours. Problems disappear overnight. Everyone becomes a minor legend in their own Slack channel.

Then fatigue walks in.

Motivation leaves quietly.

Coordination starts drifting.

Commitments made with the heart become commitments made only with the mouth.

One team races ahead while another isn’t ready for its output. Planning gets fuzzy because nobody knows when anything will actually finish. Cooperation gives way to heroics. Families, hobbies and health become “temporary” trade-offs with suspiciously recurring renewals. (Trust me on this)

And the most painful part is that extraordinary sacrifice does not guarantee extraordinary results.

The approach is difficult to control and difficult to repeat. Sometimes it works. Sometimes it doesn’t. Even success doesn’t tell you whether you can do it again.

Efficient development was less cinematic.

It was also far more interested in getting us home alive.

When Good Intentions Slow the Project Down

Then I met the classic mistakes.

They weren’t obscure failures.

They were attractive failures.

That’s worse.

Need to rescue something late? Add more people.

Need an earlier finish? Create an overly optimistic schedule.

Under pressure? Practice abandonment of planning under pressure.

Need to save time? Use short-changed upstream activities. Requirements analysis, architecture and design don’t produce code, after all. Surely we can skip them.

Still late? Try short-changed quality assurance.

Product joined in with requirements gold-plating, feature creep and developer gold-plating.

Technology brought silver-bullet syndrome, overestimated savings from new tools, and the spectacular idea of switching tools in the middle of a project.

Research-Oriented Development wanted to discover something novel while Schedule wanted predictable delivery.

Every one of them had a reasonable-sounding sales pitch.

That is why they survived.

The industry’s accumulated experience makes many of their consequences predictable: overly aggressive schedules damage planning and morale, and adding developers to an already late effort can reduce existing staff productivity through the additional coordination load.

Classic mistakes didn’t look evil.

They looked efficient.

Right up until the invoice arrived.

Does One Size Fit All? Reliability Says Absolutely Not

By then I was finally learning to distrust universal solutions.

Then Required Reliability and Extent of Distribution showed up.

Consider an inventory tracking system that loses one videotape out of a thousand.

Annoying.

Now consider heart-pacemaker control software failing at the same rate.

Not the same conversation.

The more widely distributed the software and the more important its reliability, the more carefully it has to be developed. A practice acceptable when the consequence is lost time might be reckless when human life is at stake.

So there is no single rapid-development recipe.

The more useful question becomes:

What kind of rapid development do we actually need?

A slight speed edge?
More predictability?
Better progress visibility?
Lower costs?
Maximum speed?

This is where rapid-development look-alikes started confessing.

Some people demanding speed actually wanted predictability.
Some wanted lower cost.
Some wanted assurance because previous deadlines had slipped.
Some wanted progress visibility.
Some wanted free overtime.

That last one has a tell:

The person demands a shorter schedule while refusing the feature trade-offs, resources or other support genuinely required to shorten it.

A real schedule constraint is willing to negotiate.

A fake one just keeps telling People to run faster.

Schedule Stops Pretending to Be a Date

The Probability Curve

I used to treat a planned completion date as if the project simply had to arrive there.

Schedule eventually admitted that it had been giving me probability wearing a calendar costume.

Software contains too many variables for one completion date to carry 100% certainty. Circumstances change. Developers learn about the system while building it. Some practices perform better than expected, others worse.

So completion behaves more like a range of dates.

And the curve is asymmetric.

There is an absolute limit on how quickly something can be completed.

But no equally sharp limit on how late it can become.

There are simply more ways to make software late than early.

That gives us the 50/50 break-even schedule and, around it, four uncomfortable places:

Impossible-development zone.
Rapid-development zone.
Efficient-development zone.
Slow-development zone.

The impossible-development zone was especially humbling because I had scheduled work there before.

Not intentionally, obviously.

I had simply called impossibility “ambitious”.

A project in the rapid-development zone has beaten the odds. A project in the efficient-development zone lands around a sensible balance. And a project can wander into the slow-development zone through an almost luxurious number of mistakes.

That curve changed what “late” meant to me.

Sometimes the engineering was slow.

Sometimes the expectation was fiction.

Speed Finally Has to Prove It Is Necessary

The Drop-Dead Date

Some deadlines really are hard.

A product can have a drop-dead date after which its value falls sharply.

That sounds like the perfect excuse for all-out rapid development.

Not automatically.

If efficient-development practices can finish work within Time Frame 1, before the drop-dead date, then stay efficient and focus on risk reduction.

Why?

Because some practices shorten nominal development time while increasing schedule uncertainty.

Driving faster is useless if it increases the probability that you never arrive.

Only when efficient development cannot reach the deadline, when it leaves you in Time Frame 2, do speed-oriented practices become necessary enough to justify their additional risk.

This felt backwards the first time.

The strongest schedule constraint does not automatically justify the most aggressive behaviour.

Sometimes the safest way to hit a critical date is to stop trying to be heroic.

When “Faster” Is No Longer the Question

Project Recovery

Eventually there is a darker version of this story.

Nobody knows when the software will finish.

Defects are everywhere.

The Origami Software Engineer is working 60-hour weeks through involuntary or peer-pressure-induced overtime.

Management cannot accurately determine status.

Customers have lost confidence.

The team has become defensive.

Relations between developers, marketers, managers, quality assurance and customers are strained.

Cancellation is being discussed.

Morale has hit rock bottom.

At that point the instinct is still:

How do we finish quickly?

How do we catch up?

Wrong question.

The real problem is:

How do we finish at all?

There are three fundamental recovery options: cut the software size, increase process productivity, or accept the delay and slip the schedule with damage control. Combined, they create the pragmatic fourth option:

Drop a few features, increase productivity as much as you can, and slip the schedule as needed.

Not heroic.

Real.

Recovery Means Taking Control Back

The most dangerous recovery instinct is cutting more corners.

The actual direction is the opposite.

Return to basics.

Assess your situation.

Find out whether the deadline is genuinely hard. Revisit the feature set. Apply Theory-W analysis: what does the team need to succeed, what does the customer need, and what would salvage the relationship?

Accept that the current approach is broken.

Ask the team what needs to change.

Be realistic enough to say, I don’t yet know the finish date.

Then People gets attention first.

Not “How can we motivate them more?”

If they have already been ground down, the question is how to restore the group’s morale.

Clean up major personnel problems. Clean up leadership problems. Add people carefully, if at all. Focus the time of people who already understand the system. Let steady contributors be steady contributors.

And make developers pace themselves.

You cannot sprint intelligently while pretending you don’t know how far away the finish line is.

The Project Learns to Tell the Truth Again

Miniature Milestones

Process had one last job. Make reality visible.

Create detailed miniature milestones.

Link the schedule to milestone completion.

Track schedule progress meticulously.

Record why milestones are missed.

Then recalibrate after a short time. (one or two weeks)

This is not bureaucracy returning for revenge.

This is control returning to a system that had lost it.

And only after the miniature milestone schedule has generated enough actual evidence should a meaningful new schedule be committed.

Product undergoes its own recovery:

Stabilize the requirements.
Trim the feature set.
Assess your political position.
Take out the garbage.
Get to a known good state, and build on that.
Reduce the number of defects, and keep them reduced.

Requirements stop wandering.

Defects stop reproducing faster than we can close them.

Schedule stops hallucinating.

People stop being treated as an infinitely scalable compute resource.

The strange thing is that recovery eventually teaches the same lesson that the search for speed should have taught at the beginning:

Control creates speed.

Not control as ceremony.

Control as knowing what you are building, what you are sacrificing, where you are, what can go wrong, and what “done” has enough evidence to mean.

I used to think rapid development meant forcing software to move faster.

Now I think it means removing the reasons software keeps having to go backward.

You avoid the classic mistakes. You respect development fundamentals. You manage risks. You choose schedule-oriented practices deliberately. You work through People, Process, Product and Technology instead of sacrificing one to rescue another. And when things do go wrong, you stop performing speed and start rebuilding control.

The point was never to become faster at panic.

It was to become better at finishing.

That’s not failure.
That’s evolution.

The “I liked this” Starter Pack:

Don’t let your fingers get lazy now.

Like : It tells me this was worth writing.
A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
Boost it: Especially with that one developer who definitely needs this.

Thanks for being here. It genuinely helps more than you know!

Find me elsewhere:

Professional stuff: linkedin.com/in/Aaroophan
Code stuff: github.com/Aaroophan
UI stuff: aaroophan.dev/Aaroophan
Life stuff: instagram.com/Aaroophan