DEV Community: Derek Mwale

The Economics of Running Software at Scale

Derek Mwale — Mon, 31 Aug 2026 14:09:50 +0000

Why Successful Software Eventually Becomes a Financial Problem

There is a strange moment in the life of a software product when engineering stops being only about making things work.

In the beginning, everything is simple.

You build an application.

You deploy it.

A few people sign up.

The server is mostly bored.

Your database has 37 rows.

Your cloud bill is so small that you barely notice it.

Life is beautiful.

Then something dangerous happens.

People start using your software.

At first, this feels like success.

And it is.

But success has an interesting side effect:

Successful software consumes resources.

More users mean more requests.

More requests mean more servers.

More servers mean more infrastructure.

More infrastructure means more money.

And eventually, the question changes.

It is no longer:

Can we build this?

It becomes:

Can we afford for people to use it?

That is the economics of running software at scale.

And it is one of the strangest problems in technology because, unlike many traditional businesses, software has a weird relationship with growth.

Sometimes growth makes software more profitable.

Sometimes growth destroys the business.

You can build an application that is technically brilliant, loved by millions of people, and still lose money every time someone clicks a button.

That sounds ridiculous.

But it happens.

A lot.

The Myth That Software Is Cheap

People often say software is cheap to scale.

And compared to building physical products, that is often true.

If you manufacture chairs, selling twice as many chairs means producing roughly twice as many chairs.

You need more wood.

More factories.

More transportation.

More workers.

Physical businesses usually have very obvious marginal costs.

Software appears magical because you can write something once and distribute it infinitely.

Write a mobile application once.

A million people can download it.

Write an article once.

Millions can read it.

Create an algorithm once.

It can process billions of operations.

This creates the illusion that software has zero marginal cost.

It does not.

Software has extremely low distribution costs.

That is different from having zero operating costs.

The moment software starts doing real work, costs appear everywhere.

Consider a simple API request.

A user opens an application.

The application sends a request.

That request travels through the internet.

It reaches a load balancer.

The load balancer forwards it to a server.

The server executes application code.

The application queries a database.

The database reads data from disk.

The response is returned.

Maybe something is logged.

Maybe an analytics event is generated.

Maybe a cache is updated.

Maybe an email is sent.

Maybe an AI model is called.

Maybe an image is processed.

Maybe a video is transcoded.

One innocent click can activate an entire industrial supply chain of computers.

The user sees:

"Your profile has loaded."

The infrastructure sees:

User
  |
  v
CDN
  |
  v
Load Balancer
  |
  v
Application Server
  |
  +-------> Cache
  |
  +-------> Database
  |
  +-------> Analytics
  |
  +-------> Logging
  |
  +-------> Third-Party APIs

Every arrow has a cost.

Every system has a limit.

Every limit eventually becomes somebody's invoice.

Revenue Per User vs Cost Per User

The most important equation in software economics is deceptively simple:

Profit = Revenue - Cost

But at scale, we can make it more interesting.

Imagine your application has one million users.

Each user generates an average of:

$1.00 in monthly revenue

That sounds great.

You have:

1,000,000 × $1 = $1,000,000/month

Congratulations.

You are running a million-dollar business.

Except now let's look at the cost.

Suppose each user costs:

$0.10 in compute
$0.15 in database infrastructure
$0.20 in bandwidth
$0.10 in storage
$0.15 in third-party APIs
$0.20 in customer support

Your total cost per user becomes:

$0.90

Your profit per user is:

$1.00 - $0.90 = $0.10

Your million-dollar business is now making:

$100,000 in gross operating margin

Still not bad.

But now imagine your costs increase slightly.

Perhaps AI API usage becomes more expensive.

Perhaps bandwidth grows.

Perhaps users upload more videos.

Perhaps your database becomes difficult to scale.

Now the cost per user becomes:

$1.20

You are now losing:

$0.20 per user

With one million users:

1,000,000 × $0.20 = $200,000/month

And suddenly growth is no longer your friend.

Every new customer makes the company poorer.

This is one of the most terrifying economic situations in software.

You can have:

More users
More downloads
More engagement
More attention
More servers

And less money.

The Most Dangerous Sentence in Technology

There is a sentence that has probably destroyed an incredible amount of money in the technology industry.

It goes like this:

"Let's worry about scaling later."

Sometimes this is good advice.

You should not build a global distributed database for your first ten users.

That would be ridiculous.

But there is another extreme.

Some companies build products without understanding their unit economics.

They know how much money they make.

They have absolutely no idea how much an individual user costs.

This becomes especially dangerous with:

AI products
Video platforms
Cloud storage
Free APIs
Gaming platforms
Messaging applications
Real-time systems

Imagine creating an AI writing application.

You charge users:

$10/month

Sounds simple.

But your users are heavy users.

Each one consumes:

$18/month

in AI inference.

You have created a very successful mechanism for transferring money from your company to your users.

The more subscribers you get, the faster you lose money.

This is why pricing is not just a business problem.

Pricing is a systems engineering problem.

Every Feature Has an Economic Architecture

Engineers often think about architecture in technical terms.

We ask:

Is it scalable?
Is it reliable?
Is it maintainable?
Is it fast?

But there is another question:

How expensive is this architecture?

Consider two systems.

System A

User
  |
  v
Monolith
  |
  v
PostgreSQL

Simple.

Cheap.

Easy to understand.

Now consider System B.

User
  |
  v
Global CDN
  |
  v
API Gateway
  |
  +--------+
  |        |
  v        v
Auth     Payments
Service  Service
  |
  +--------+
  |
  v
Message Queue
  |
  v
Worker Cluster
  |
  +--------+
  |        |
  v        v
Redis   Kafka
  |
  v
Distributed Database

Technically impressive.

Possibly necessary.

Possibly completely insane.

The important question is not:

Which architecture looks more advanced?

The question is:

Which architecture produces more value than it costs?

A complicated system can solve problems you do not have.

And every unnecessary system creates:

Infrastructure costs
Monitoring costs
Developer costs
Debugging costs
Operational costs
Cognitive costs

Software architecture is often an economic trade-off disguised as an engineering decision.

The Cloud Changed the Economics of Starting Companies

Before cloud computing, starting an internet company was expensive.

You needed servers.

Actual servers.

Physical machines.

You had to buy them.

Install them.

Put them somewhere.

Connect them to networks.

Cool them.

Maintain them.

Then cloud computing arrived and changed everything.

Suddenly, you could start with almost nothing.

You could launch a startup using infrastructure from:

Instead of buying a server for thousands of dollars, you could rent computing power by the hour.

This was revolutionary.

But it introduced a different economic model.

The cloud transformed infrastructure from a capital expense into an operating expense.

Instead of saying:

We need $50,000 to buy servers.

You could say:

We need $50 this month.

That is fantastic when you are small.

But there is a strange phenomenon that happens at scale.

The same flexibility that makes the cloud cheap for startups can become expensive for large companies.

You start with:

$20/month

Then:

$200/month

Then:

$2,000/month

Then:

$20,000/month

And one day somebody opens the billing dashboard and discovers that the application is consuming the GDP of a small village.

The cloud is not expensive.

The cloud is extremely good at making you capable of spending money.

Compute Is the New Electricity

Modern software consumes computation the way industrial machines consume electricity.

Every feature requires energy.

Every calculation requires CPU cycles.

Every database query requires work.

Every AI request requires enormous computational resources.

This becomes easier to understand if we think of software as a factory.

Your application receives raw materials:

Requests
Data
Images
Videos
Events
Messages

Your infrastructure processes them.

Then it produces:

Responses
Predictions
Files
Recommendations
Transactions

The factory is invisible.

But it is still a factory.

And factories consume resources.

A useful mental model is:

User Demand
     |
     v
Infrastructure Work
     |
     v
Resource Consumption
     |
     v
Financial Cost

The engineering challenge is reducing the amount of work required to produce value.

That is why performance optimization can sometimes become a business strategy.

If you make an API twice as efficient, you might need fewer servers.

If you reduce database queries, you might reduce infrastructure costs.

If caching eliminates expensive computations, your margins improve.

A performance improvement can literally become profit.

Why Caching Is Basically Economic Magic

Caching is one of the most beautiful ideas in computer science.

Instead of repeatedly doing expensive work, you remember the answer.

Imagine a million users requesting the same article.

Without caching:

1,000,000 users
        |
        v
1,000,000 database queries

With caching:

Database
    |
    v
Cache
    |
    +--------------------+
    |                    |
    v                    v
User                 User
User                 User
User                 User

Instead of performing the same expensive operation a million times, you might perform it once.

That is not just a technical optimization.

It changes the economics.

Suppose a database query costs infrastructure resources.

Without caching:

Cost = High

With caching:

Cost = Much Lower

The product did not change.

The user experience might actually improve.

But your infrastructure bill changes dramatically.

Caching is one of the rare situations where:

Users are happier
Servers are happier
Databases are happier
Accountants are happier

Computer science occasionally gives everyone a win.

Databases Are Where Your Money Goes to Become Philosophy

Databases are fascinating because they usually behave perfectly when you are small.

Your application has:

10 users

Everything is fast.

Then:

1,000 users

Still fine.

Then:

100,000 users

Interesting.

Then:

10,000,000 users

Now you are attending meetings about sharding.

Nobody is happy.

At scale, databases become expensive because data has physical properties.

Data must be:

Stored
Indexed
Replicated
Backed up
Transferred
Secured

The more data you collect, the more infrastructure you need.

And modern companies collect ridiculous amounts of data.

Sometimes companies store information because:

"We might need it someday."

This is how you accidentally build a digital warehouse filled with things nobody uses.

Data has an economic cost.

If you store one gigabyte of unnecessary data, that may seem insignificant.

Store petabytes of unnecessary data, replicate it globally, back it up, index it, and suddenly "just in case" becomes a line item on a financial report.

A good engineering principle is:

The cheapest data is the data you never needed to store.

Bandwidth: The Invisible Tax

Bandwidth is one of the easiest costs to ignore.

Until it isn't.

Text is cheap.

Images are larger.

Video is enormous.

Imagine a website serving a 5 MB image.

That does not sound particularly large.

Now imagine:

5 MB × 1,000,000 views

You have now transferred approximately:

5,000,000 MB

Or around:

5 TB

For one image.

Now imagine a video platform.

Suddenly, your business is moving data around the planet like a logistics company.

This is why companies use:

Compression
CDNs
Adaptive streaming
Image optimization
Modern file formats

The user thinks:

"The video plays."

The engineer thinks:

"How many gigabytes did that just cost?"

AI Has Made Unit Economics Even Stranger

Artificial intelligence has introduced a fascinating problem into software economics.

Traditional software is often cheap to run after it has been built.

AI software can be different.

A normal calculator application can perform millions of calculations cheaply.

But an AI application might perform extremely expensive inference for every user interaction.

This creates a new relationship between usage and cost.

Consider a traditional SaaS product.

A user pays:

$20/month

They use the application heavily.

The additional infrastructure cost might be relatively small.

Now consider an AI product.

The user pays:

$20/month

But every prompt requires expensive model inference.

The user generates:

Articles
Images
Code
Videos
Audio

Suddenly, the user's behavior directly affects your infrastructure cost.

The most active customers might be your least profitable customers.

That is a very strange business model.

Imagine owning a restaurant where your best customers eat everything and somehow send you the bill afterward.

AI companies therefore have to think carefully about:

Usage limits
Token economics
Subscription pricing
Model routing
Caching
Smaller models
Rate limits

The future of AI may not only be about building smarter models.

It may also be about building economically intelligent systems.

A system that knows when to use an expensive model.

When to use a cheap model.

When to cache.

When to reject unnecessary computation.

The smartest AI system may not be the one that uses the most intelligence.

It may be the one that knows when intelligence is worth paying for.

The Cost of Reliability

Reliability sounds universally good.

Of course you want 99.999% uptime.

Nobody wants their application to go down.

But reliability costs money.

Imagine running one server.

Server

Cheap.

But if it fails:

Application = Dead

Now add redundancy.

Server A
    |
    +----> Application
    |
Server B

Better.

But now you are paying for additional infrastructure.

Add multiple regions.

Africa Region
Europe Region
US Region
Asia Region

Now your application can survive regional failures.

Wonderful.

Your finance department has stopped speaking to you.

High availability is essentially insurance.

You pay for infrastructure that you hope you will not need.

That does not mean it is wasteful.

It means reliability must be proportional to business value.

A banking system may justify extremely expensive redundancy.

A personal blog probably does not.

This is why the phrase:

"Five nines of availability"

should always be followed by:

"How much does that cost?"

Scaling Is Not Free, Even When It Is Automatic

Modern cloud platforms allow something magical:

Auto-scaling.

Traffic increases.

New servers appear.

Traffic decreases.

Servers disappear.

Technically, this is fantastic.

Economically, it can be terrifying.

Imagine your application goes viral.

Your traffic increases by:

1000%

Your infrastructure automatically scales.

Your application stays online.

Everybody celebrates.

Then the bill arrives.

Congratulations.

You survived success.

Now you cannot afford it.

This is why scaling policies need economic limits.

A system should not simply ask:

Can we handle more traffic?

It should also ask:

Can we afford to handle more traffic?

Rate limiting is therefore not always about protecting the server.

Sometimes it protects the business.

Free Users Are Not Free

One of the biggest mistakes in software is misunderstanding free users.

A free user may not pay money.

But they still consume:

Storage
Compute
Bandwidth
Support
Database capacity

The company may call them:

Users.

The infrastructure team sees:

Expenses.

This does not mean free users are bad.

Free tiers can be incredibly valuable.

They can generate:

Growth
Network effects
Brand awareness
Future customers

But free usage must have an economic strategy.

You need to understand:

Free User Cost
        |
        v
Conversion Probability
        |
        v
Future Revenue

If free users cost very little and some eventually become paying customers, the model can work beautifully.

If free users are extremely expensive and rarely convert, you have a problem.

The internet is full of companies discovering this problem several million dollars later.

Engineering Decisions Are Business Decisions

One of the most important lessons you learn as software becomes larger is that engineering cannot be separated completely from economics.

Consider these choices:

Use a faster algorithm?

Potentially fewer servers.

Add caching?

Potentially lower database costs.

Compress images?

Potentially lower bandwidth costs.

Delete unused data?

Potentially lower storage costs.

Optimize a database query?

Potentially lower infrastructure requirements.

Reduce API calls?

Potentially lower third-party costs.

These look like engineering decisions.

But they affect:

Profit Margin

This is why great engineers eventually develop economic intuition.

They begin asking questions like:

What does this operation cost at one million requests?

That is a different mindset from:

Does it work?

At scale, "works" is only the beginning.

The Cost of Human Beings

Infrastructure is expensive.

But people are often more expensive.

A complex distributed system might require:

Backend engineers
DevOps engineers
Site reliability engineers
Security engineers
Database engineers

Imagine two architectures.

Architecture One

Requires:

3 engineers

Architecture Two

Requires:

15 engineers

But Architecture Two saves:

$2,000/month

in cloud infrastructure.

You may have just spent hundreds of thousands of dollars in salaries to save twenty-four thousand dollars per year.

This is why optimization must be economically rational.

The cheapest server is not always worth the most expensive engineering team.

Sometimes the best solution is simply:

Pay the cloud bill.

This can be surprisingly difficult for engineers to accept because engineers love optimization.

I understand.

I am one of them.

We can spend three days optimizing something to save $4.

It feels amazing.

Financially, perhaps questionable.

The Weird Mathematics of Scale

Scale creates nonlinear behavior.

This is one of the reasons it is difficult.

You might think:

10× users = 10× cost

Sometimes.

But sometimes costs increase faster.

For example:

More Users
    |
    v
More Database Connections
    |
    v
Database Contention
    |
    v
Slower Queries
    |
    v
More Servers Waiting
    |
    v
Even Higher Cost

Or the opposite can happen.

You might get economies of scale.

More Users
    |
    v
Shared Infrastructure
    |
    v
Lower Cost Per User

This is why scale is not simply about adding machines.

It is about understanding how the entire system behaves under pressure.

A tiny inefficiency becomes enormous when repeated millions of times.

Imagine a function that wastes:

1 millisecond

Nobody cares.

Now execute it:

1 billion times

You have wasted:

1 billion milliseconds

Which is over:

11 days

of computational work.

Scale turns small mistakes into companies.

Sometimes very expensive companies.

The Cheapest Architecture Is Often the Boring One

Technology culture sometimes makes boring software sound embarrassing.

Nobody wants to say:

We built a simple monolith with PostgreSQL and Redis.

That does not sound like a conference keynote.

But boring architecture has advantages.

It is often:

Easier to understand
Easier to maintain
Easier to debug
Cheaper to operate
Easier to hire for

A boring architecture can be economically superior to an exciting one.

This is not an argument against advanced systems.

Sometimes you genuinely need:

Distributed systems
Microservices
Event streaming
Global replication

But complexity should be purchased when necessary.

Not because it looks impressive.

A simple system that costs $500/month is often better than a sophisticated system that costs $15,000/month to solve problems you do not have.

Software Is a Machine That Converts Money

Eventually, every large software company can be understood as a machine.

Money enters.

Computers perform work.

Users receive value.

Hopefully, more money comes back.

The machine looks something like this:

Capital
   |
   v
Engineers
Infrastructure
Software
   |
   v
Product
   |
   v
Users
   |
   v
Revenue
   |
   +----------------+
   |                |
   v                v
Profit          More Growth

The goal is not simply to minimize costs.

That would be a terrible strategy.

You could turn off all the servers.

Congratulations.

Your infrastructure costs are now zero.

Your revenue is also probably zero.

The goal is to maximize:

Value created per unit of cost.

That is the real economics of software.

The Future Will Be Written in Efficiency

For years, technology companies were rewarded primarily for growth.

More users.

More downloads.

More attention.

More engagement.

But as infrastructure becomes more expensive—and as AI makes computation a direct cost of delivering products—efficiency is becoming strategically important again.

The companies of the future may compete on:

Better algorithms
Cheaper inference
More efficient infrastructure
Better hardware
Smarter caching
Lower energy consumption

In other words:

Computer science is becoming economics again.

An algorithm is no longer just academically elegant.

A better algorithm can reduce costs.

A better architecture can improve margins.

A better database design can delay millions of dollars in infrastructure spending.

A better caching strategy can transform an unprofitable product into a profitable one.

The engineers who understand both systems and economics will become incredibly valuable.

Because they will not only know how to build things.

They will know how to build things that can survive success.

The Final Lesson

The most dangerous moment for a software product is not always failure.

Sometimes it is growth.

Failure is obvious.

Nobody uses your application.

The servers are quiet.

The bill is small.

You know you have a problem.

Growth can be more deceptive.

Users arrive.

Traffic increases.

Your graphs go up.

Your application becomes popular.

And somewhere, quietly, your infrastructure costs begin climbing faster than your revenue.

That is why running software at scale is not simply an engineering problem.

It is a balancing act between:

Performance
Reliability
Cost
Growth
Complexity
Revenue

You cannot maximize everything.

You have to make trade-offs.

The best software systems are not necessarily the fastest.

Or the most distributed.

Or the most technologically impressive.

They are the systems that create enough value to justify the resources they consume.

That is the strange economics of software.

You write code.

The code runs on computers.

The computers consume resources.

The resources cost money.

The money determines what the company can survive.

And eventually, somewhere between a database query and a cloud invoice, you discover something important:

Software is not just code.

Software is an economic machine.

Every request has a cost.

Every feature has a price.

Every architectural decision has financial consequences.

And at scale, the companies that survive are often not the ones with the most powerful systems.

They are the ones that understand the economics hiding underneath every line of code.

Because the real challenge of scaling software is not merely answering:

Can the system handle ten million users?

The harder question is:

Can the business afford ten million users?

And that question may be one of the deepest engineering problems of all.

The Weird Economics of AI Tokens

Derek Mwale — Mon, 31 Aug 2026 14:02:57 +0000

Why the Smallest Unit of Artificial Intelligence Might Become One of the Most Important Economic Units of the Internet

There is something deeply strange happening in computing.

For decades, software economics was relatively easy to understand.

You bought a computer.

You bought some software.

Maybe you paid for a server.

Maybe you paid monthly for a SaaS subscription.

The economics were imperfect, but the mental model was simple.

You paid for things.

You paid for storage.

You paid for bandwidth.

You paid for CPU time.

You paid for users.

Then artificial intelligence arrived and decided that none of that was weird enough.

Now we are increasingly paying for tokens.

Not cryptocurrency tokens.

Not arcade tokens.

Not authentication tokens.

AI tokens.

Tiny fragments of language that have somehow become the economic meter of intelligence.

You type a sentence.

A machine breaks it into pieces.

Those pieces travel through billions of parameters sitting inside expensive GPU clusters.

Electricity flows.

Memory moves.

Silicon gets hot.

And somewhere, an accounting system quietly records:

1,847 input tokens.
923 output tokens.

Congratulations.

You have just purchased a small amount of artificial thought.

That sentence sounds ridiculous.

But economically, that is increasingly what modern software is becoming.

AI companies are turning data centers into factories.

The raw materials are electricity, silicon, memory bandwidth, and capital.

The product is tokens.

And developers are buying those tokens like factories once bought electricity.

That is where things become interesting.

Because AI tokens have some of the strangest economics we have ever seen in technology.

A token can cost almost nothing and still generate enormous value.

A cheaper token can make your product more expensive.

A more expensive model can reduce your total costs.

The price of intelligence is falling rapidly, while the amount of money being spent on intelligence is exploding.

And perhaps strangest of all:

The better AI becomes at thinking, the more difficult it becomes to understand what you are actually paying for.

Welcome to the weird economics of AI tokens.

1. What Exactly Are We Buying?

Let us start with the obvious question.

What is a token?

A token is not exactly a word.

It is not exactly a character either.

It is a chunk of text that an AI model processes.

The sentence:

Artificial intelligence is changing software.

might be broken into several tokens depending on the tokenizer used by the model.

Some common words might be represented as single tokens.

Longer or unusual words might be broken into multiple pieces.

Punctuation can become tokens.

Spaces can matter.

Code behaves differently.

Languages behave differently.

The word “computer” may be cheap in one context.

A complex Rust function with generics, macros, lifetimes, and deeply nested types might become a much larger computational meal.

But from the perspective of an AI provider, tokens solve an important problem.

They create a unit of measurement.

Electricity has kilowatt-hours.

Internet providers have gigabytes.

Cloud platforms have CPU-hours.

AI has tokens.

That means an API provider can say:

Give us some text.
We will process it.
We will generate more text.
And we will charge you based on how much language moved through the machine.

It sounds almost elegant.

But this simplicity is deceptive.

Because not all tokens are economically equal.

A token entering a model is not necessarily equivalent to a token leaving it.

A token generated at the end of a long reasoning process may have required dramatically more computation than a simple input token.

A token generated by a small model and a token generated by a frontier reasoning model may represent completely different amounts of infrastructure.

The meter is simple.

The machinery behind the meter is not.

Recent analyses of AI inference economics increasingly make this distinction: token pricing is useful as a billing abstraction, but the price per token does not directly tell you the cost per useful outcome.

That is the first weird thing about AI tokens.

The unit we pay for is not necessarily the unit that creates value.

2. A Token Is Not a Thought

Imagine two AI requests.

Request A:

What is the capital of Zambia?

The AI responds:

Lusaka.

Request B:

Design a fault-tolerant distributed payment system capable of processing ten million transactions per day, explain the architecture, identify failure modes, generate database schemas, and provide implementation examples.

Both requests produce tokens.

But pretending those tokens have identical economic value would be absurd.

One answer saves you five seconds of Googling.

The other might save a team of engineers weeks of work.

This creates an important distinction:

Token consumption is not the same thing as economic value.

Suppose an AI system consumes one million tokens.

What did those tokens accomplish?

Did they generate spam?

Did they summarize documents nobody needed?

Did they produce a brilliant legal analysis?

Did they help write software?

Did they prevent a million-dollar operational failure?

The token count alone cannot tell us.

This is one of the biggest problems with AI economics.

We have a very precise meter for measuring usage.

But we still have a terrible meter for measuring usefulness.

Imagine if your electricity company charged you based on kilowatt-hours but nobody could tell you whether those kilowatt-hours powered a hospital, a factory, or a forgotten lightbulb.

That is roughly where AI is today.

We can measure:

tokens consumed,
tokens generated,
GPU time,
latency,
API costs.

But measuring:

intelligence gained,
productivity created,
decisions improved,
mistakes avoided,
revenue generated,

is much harder.

This means companies can become obsessed with reducing token costs while completely missing the larger question:

Are these tokens doing anything useful?

A $100 AI request that saves a company $10,000 is incredibly cheap.

A $0.01 AI request that nobody uses is infinitely expensive.

This is why the cheapest model is not automatically the most economical model.

A model that costs more per token but solves the task correctly may actually be cheaper than a weak model that produces bad answers, creates retries, requires human corrections, and causes downstream errors. That distinction between token price and cost per useful outcome is increasingly central to production AI economics.

3. The Input Token vs Output Token Mystery

Now things get even stranger.

Most AI APIs charge differently for input and output tokens.

Why?

Because they are not computationally equivalent.

Input tokens are often processed in a stage commonly associated with prefill.

The model receives a large chunk of context and processes it.

This work can be parallelized efficiently.

Output generation is different.

The model produces one token.

Then another.

Each token depends on what came before.

It is a sequential process.

The AI cannot simply generate the entire paragraph instantly in the same way it processes a batch of input text.

That makes output generation economically different.

You could think of it like this:

INPUT

User Prompt
     │
     ▼
[ Process many tokens ]
     │
     ▼
Model Context


OUTPUT

Token 1
   │
   ▼
Token 2
   │
   ▼
Token 3
   │
   ▼
Token 4
   │
   ▼
...

The second process is more annoying for hardware.

The GPU must repeatedly perform work, move data, and maintain state.

Memory bandwidth becomes incredibly important.

This is one reason output tokens are often priced significantly higher than input tokens.

The strange part is that users often think about AI costs backwards.

They worry about what they type.

But the expensive part might be what the AI decides to say back.

You write:

Explain quantum computing.

That might be a tiny prompt.

The AI responds with 4,000 words.

Congratulations.

You just asked a machine to write an essay.

And from an economic perspective, the machine probably did much more work generating the answer than processing your original question.

This creates a funny paradox.

The shortest user can sometimes be the most expensive user.

The person typing:

Explain everything.

may be more expensive than the person uploading a carefully structured document.

Recent technical analyses of inference economics describe this asymmetry as a consequence of the different hardware behavior of prefill and autoregressive decoding, with output generation often being more expensive because of its sequential and memory-intensive nature.

4. The Cheapest AI Can Make You Poor

This is where startups begin learning painful lessons.

Imagine two models.

Model A

Costs:

$1 per million tokens

Model B

Costs:

$10 per million tokens

The obvious answer seems simple.

Use Model A.

It is ten times cheaper.

But let's imagine what happens.

Model A frequently misunderstands customer requests.

Your system has to retry.

Sometimes it generates invalid JSON.

Your application asks it again.

Sometimes it chooses the wrong tool.

Your agent repeats the workflow.

Your support team manually fixes the result.

The customer becomes frustrated.

Now look at Model B.

It gets the answer right the first time.

It uses fewer retries.

It calls fewer tools.

It generates cleaner output.

Suddenly:

MODEL A

Cheap Token
    │
    ▼
Bad Result
    │
    ▼
Retry
    │
    ▼
Another Retry
    │
    ▼
Human Fix
    │
    ▼
Expensive Outcome


MODEL B

Expensive Token
    │
    ▼
Correct Result
    │
    ▼
Cheap Outcome

This is one of the most important lessons in AI architecture.

Cost per token is not cost per task.

And cost per task is not even cost per outcome.

A cheap model that needs ten attempts is not cheap.

A powerful model that solves the problem immediately may be.

The economics become even stranger with AI agents.

A normal chatbot might answer once.

An agent might:

Think.
Call a tool.
Read the result.
Think again.
Call another tool.
Analyze the response.
Retry something.
Generate a final answer.

Suddenly one user request becomes dozens of model calls.

One request becomes an entire miniature economy of tokens.

This is why agentic software changes the cost structure of AI products so dramatically. Token usage can multiply through retrieval, tool calls, retries, and multi-step workflows, making request count a poor proxy for actual cost.

5. The Infinite Appetite Problem

Normally, when something becomes cheaper, people spend less money buying it.

AI does not behave like that.

AI behaves more like gasoline, roads, or internet bandwidth.

As the cost of intelligence falls, people find more things to do with intelligence.

Imagine AI costs $100 per task.

You use it for extremely important tasks.

Maybe medical research.

Maybe financial analysis.

Maybe enterprise software.

Now imagine the cost falls to $1.

Suddenly you use it for:

writing emails,
generating product descriptions,
summarizing meetings,
translating conversations,
writing code,
analyzing spreadsheets,
answering customer questions,
generating game dialogue,
reviewing documents,
creating personalized education.

Now imagine it falls to one cent.

You stop asking:

Should we use AI?

And start asking:

Why isn't everything using AI?

That is the strange economic force behind falling token prices.

Lower prices can increase total spending.

This is closely related to the famous economic idea known as Jevons paradox.

When a resource becomes more efficient to use, people may consume more of it rather than less.

Cars became more fuel-efficient.

People drove more.

Computers became cheaper.

We bought billions of them.

Bandwidth became cheaper.

We invented 4K video streaming.

Storage became cheaper.

We started keeping everything.

AI tokens are likely heading in the same direction.

As intelligence becomes cheaper, software will consume more intelligence.

The result may be bizarre.

The price of an individual token could fall dramatically.

But the total number of tokens consumed could grow even faster.

This means:

Cost Per Token ↓↓↓

Usage Per Person ↑↑↑↑↑↑↑↑↑↑

Total Spending ?

Nobody knows exactly where that equation settles.

But the possibility is fascinating.

AI might become cheaper and more expensive at the same time.

Cheaper per unit.

More expensive in total.

Recent market analysis has highlighted exactly this tension: per-token costs have fallen sharply, while overall demand for tokens continues to surge as cheaper inference unlocks more use cases.

6. Context Windows Are Becoming Economic Weapons

Modern AI models can remember enormous amounts of information.

You can feed them documents.

Books.

Codebases.

Meeting histories.

Entire databases.

And this creates another strange economic problem.

Memory costs money.

Imagine your AI application has a conversation with a customer.

Every new message includes:

System Prompt

Conversation History

User Preferences

Retrieved Documents

Tool Results

New User Message

Every single time.

The user asks:

Hi.

Your application quietly sends:

20,000 tokens of context.

The AI replies:

Hello! How can I help?

The customer sees a friendly chatbot.

Your infrastructure sees a financial crime scene.

This is the hidden economics of AI applications.

The visible interaction might be tiny.

The invisible context might be enormous.

A chatbot is not simply responding to the latest message.

It may be dragging its entire past through the model every time.

Then developers add RAG.

Now the system retrieves documents.

Maybe five documents.

Maybe ten.

Each document contains thousands of tokens.

The user asks a simple question.

The system sends an entire library.

The user receives three sentences.

This is like hiring a professor to read five books every time someone asks:

What time does the shop close?

That is why AI architecture increasingly becomes economic architecture.

Good RAG is not simply about finding more information.

It is about finding just enough information.

Too little context creates bad answers.

Too much context creates expensive answers.

There is an economic optimization problem hiding inside every prompt.

A useful way to think about it is:

MORE CONTEXT
     │
     ├──► Potentially Better Answer
     │
     └──► Higher Cost


LESS CONTEXT
     │
     ├──► Lower Cost
     │
     └──► Potentially Worse Answer

The future AI engineer may spend as much time managing information economics as writing prompts.

7. Data Centers Are Becoming Token Factories

This might be the biggest conceptual shift of all.

We used to think about data centers as places where computers lived.

Servers processed websites.

Databases stored information.

APIs handled requests.

But increasingly, data centers are becoming factories.

Their product is not a physical object.

Their product is inference.

Or more specifically:

useful tokens produced at scale.

Think about a traditional factory.

It has:

raw materials,
machines,
energy,
labor,
production capacity,
supply chains,
operating costs.

Now look at an AI data center.

It has:

electricity,
GPUs,
memory,
networking,
cooling,
engineers,
model weights,
production capacity.

The output is tokens.

NVIDIA and other infrastructure players increasingly frame AI inference in precisely this industrial language: data centers can be viewed as “token factories,” where infrastructure converts energy and hardware capacity into AI output.

This is a profound idea.

Because it means the economics of AI may start looking less like traditional software economics and more like industrial economics.

Capacity matters.

Utilization matters.

Idle hardware matters.

Production efficiency matters.

Imagine buying a billion-dollar factory.

Then only using it 10% of the time.

That would be terrible.

The same thing happens with AI infrastructure.

A GPU sitting idle is expensive.

A GPU generating useful work is productive.

This is why batching matters so much.

If one GPU can efficiently process many requests together, the cost per token can fall dramatically.

If every request gets its own lonely GPU moment, economics become ugly.

The future AI war might not simply be:

Who has the smartest model?

It might also be:

Who can manufacture intelligence most efficiently?

8. The Subscription Illusion

Here is another weird thing.

Many consumers pay a flat monthly subscription for AI.

Maybe $20.

Maybe $30.

Maybe more.

Then they use the AI constantly.

From the user's perspective:

AI is unlimited.

From the provider's perspective:

Absolutely not.

Every message costs something.

Every image costs something.

Every generated file costs something.

Every reasoning task consumes infrastructure.

So subscription AI creates an interesting economic mismatch.

The user sees:

Monthly Price = Fixed

The provider sees:

Cost = Variable

That is dangerous.

Imagine a gym.

You pay a monthly membership.

The gym hopes you do not visit twelve times per day.

AI subscriptions have the same problem.

Most users might use AI moderately.

A small group might become extremely heavy users.

Those users can consume enormous amounts of compute.

This means AI companies must constantly balance:

subscriptions,
API pricing,
usage limits,
model routing,
cheaper models,
expensive models,
infrastructure capacity.

In other words:

AI companies are partly becoming insurance companies.

They are betting that average usage will remain economically manageable.

Heavy users are subsidized by light users.

Efficient models subsidize expensive models.

Some requests are profitable.

Some are not.

The entire business becomes a portfolio of computational risk.

And the more capable AI becomes, the more unpredictable that risk may become.

9. Reasoning Makes Everything Worse

Then reasoning models arrived.

And they made token economics even stranger.

Traditional AI might receive a prompt and produce an answer.

Reasoning AI might perform significantly more internal computational work before producing the final answer.

The user sees:

The answer is 42.

But behind the scenes, the system may have consumed vastly more computational resources than the visible answer suggests.

This creates a bizarre situation.

The final response may be tiny.

The computational journey may be enormous.

A human might say:

That was a short answer.

The infrastructure might say:

You have no idea what we went through.

Reasoning models destroy the old intuition that longer output equals higher cost.

A short answer might require a massive amount of computation.

A long answer might be relatively easy.

The economics of AI become less visible.

The customer sees the final product.

The infrastructure sees the entire production process.

This introduces an uncomfortable question:

Should customers pay for thinking?

Imagine hiring a consultant.

You pay them for the result.

Not for every thought they had while driving to the office.

AI complicates this.

If a model needs more computation to solve a difficult problem, somebody has to pay.

Maybe the provider absorbs it.

Maybe the customer pays indirectly.

Maybe the system switches models.

Maybe the model is rate-limited.

Maybe reasoning becomes a premium feature.

The market is still figuring this out.

Research on AI tokenomics has increasingly emphasized that visible token usage and the actual computational work behind reasoning systems can diverge significantly, making simple token accounting an incomplete picture of value and cost.

10. AI Is Creating a New Type of Software Budget

Traditional software teams ask questions like:

How many servers do we need?

How much storage will we consume?

How many users can our database handle?

AI teams now ask:

How many tokens will this feature consume?

This sounds innocent.

Until you realize nobody knows.

A normal API request might have predictable costs.

A database query is usually somewhat measurable.

But an AI conversation can vary wildly.

One user says:

Hello.

Another uploads a 400-page PDF.

Another asks the AI to analyze it.

Another asks for ten revisions.

Another starts an agent.

The agent loops.

The agent calls tools.

The agent retries.

The agent gets confused.

Your cloud bill becomes a philosophical document.

This is why AI products need something close to token observability.

Developers need dashboards showing:

tokens per user,
tokens per feature,
tokens per conversation,
tokens per successful task,
tokens wasted on retries,
tokens consumed by tools,
tokens consumed by context,
tokens consumed by failed requests.

Because eventually, someone in finance will ask:

Why did our AI bill triple this month?

And “people used the chatbot more” will not be a sufficient answer.

The real answer might be:

We introduced a feature that added 8,000 tokens of context to every request.

Or:

The agent entered a retry loop.

Or:

We used the frontier model for everything.

Or:

One customer discovered how to ask the AI to write novels.

The economics of AI will increasingly force software engineers to think like economists.

11. The Future Might Be Model Routing

One of the most sensible solutions to token economics is surprisingly simple.

Stop using the same model for everything.

Imagine a company with three AI models.

Cheap Model

Fast.

Good enough.

Used for simple tasks.

Medium Model

More capable.

Used for moderate complexity.

Frontier Model

Expensive.

Extremely capable.

Used only when necessary.

Now imagine an AI router:

                USER REQUEST
                     │
                     ▼
              Complexity Check
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       SIMPLE     MEDIUM      HARD
          │          │          │
          ▼          ▼          ▼
      Cheap AI    Mid AI    Frontier AI

This is likely where much of AI economics is heading.

Not one giant model.

A mixture of intelligence.

The system decides how much intelligence the problem deserves.

That is a fascinating concept.

In the future, software may dynamically allocate intelligence the way operating systems allocate CPU resources.

A simple question receives cheap intelligence.

A difficult engineering problem receives expensive intelligence.

A critical legal task receives the best available intelligence.

This means the architecture itself becomes an economic engine.

The application is no longer simply:

User → AI

It becomes:

User
  │
  ▼
Understand Task
  │
  ▼
Estimate Complexity
  │
  ▼
Choose Intelligence Level
  │
  ▼
Spend Appropriate Tokens
  │
  ▼
Return Result

That might be one of the defining design patterns of AI software.

Not just prompt engineering.

Intelligence allocation.

12. The Most Important Metric Might Not Be Tokens

Eventually, I think the industry will become slightly embarrassed by how obsessed it was with token prices.

Because tokens are a means.

Not the end.

Nobody really wants a million tokens.

Nobody wakes up and says:

I hope my startup consumes 400 billion tokens this quarter.

What they want is:

customers served,
code written,
documents analyzed,
diseases detected,
fraud prevented,
lessons personalized,
businesses automated.

Tokens are simply the fuel.

And this is where AI economics might mature.

Instead of measuring:

Cost per million tokens

companies may increasingly measure:

Cost per useful task.

Or even:

Cost per successful outcome.

Imagine two systems.

System A:

$0.02 per task

70% success rate

System B:

$0.50 per task

99% success rate

Which one is cheaper?

The answer depends on the cost of failure.

If failure is harmless, System A may be excellent.

If failure means losing a customer, System B may be dramatically cheaper.

This is the central economic problem of AI.

Intelligence cannot be priced properly without understanding the value of being correct.

And correctness depends on context.

The value of an AI-generated joke is different from the value of an AI-generated medical insight.

The value of autocomplete is different from the value of preventing a financial disaster.

One token might be worthless.

Another might save millions.

Same unit.

Completely different economics.

13. The Final Paradox

The weirdest thing about AI tokens is that they might eventually become invisible.

Right now developers talk about them constantly.

Input tokens.

Output tokens.

Context windows.

Millions of tokens.

Token budgets.

Token optimization.

But electricity used to be complicated too.

Nobody wants to think about voltage while turning on a lamp.

Cloud infrastructure used to be complicated.

Most developers no longer think about physical servers when deploying a website.

Eventually, AI intelligence might become infrastructure.

Software will simply ask:

How much intelligence do I need?

And somewhere underneath the application:

models will compete,
routers will select providers,
systems will optimize context,
caches will reuse computation,
GPUs will manufacture tokens,
infrastructure will manage cost.

The token might disappear from the developer's mental model.

But its economics will remain underneath everything.

And that is why AI tokens matter.

They are not just a billing mechanism.

They are the bridge between:

language and electricity.

Between:

human questions and silicon.

Between:

intelligence and economics.

For the first time in computing history, we are beginning to treat something resembling intelligence as a metered resource.

You can buy more of it.

You can optimize it.

You can waste it.

You can route it.

You can budget it.

You can manufacture it in enormous factories.

And one day, perhaps, companies will manage intelligence the same way they manage electricity.

They will ask:

How much do we need?

How efficiently can we produce it?

What is it worth?

That is the weird economics of AI tokens.

A token is tiny.

Sometimes it is only part of a word.

But somewhere inside that tiny fragment is an enormous chain of events.

Electricity was generated.

A data center was built.

A GPU was manufactured.

Memory moved billions of times.

A model performed mathematical operations.

A machine predicted the next piece of language.

And eventually, one tiny token appeared on your screen.

You read it.

Maybe it helped you.

Maybe it changed your decision.

Maybe it saved you time.

Maybe it made you money.

Maybe it did absolutely nothing.

And that is the strange part.

The infrastructure knows exactly how many tokens you consumed.

But the world is still trying to figure out what they were worth.

Conclusion: Intelligence Is Becoming a Utility

For most of human history, intelligence was something you had to hire.

You hired experts.

Engineers.

Lawyers.

Teachers.

Analysts.

Writers.

Programmers.

Now we are slowly entering a world where some forms of intelligence can be accessed through an API.

Metered.

Programmable.

Scalable.

And increasingly cheap.

But cheap intelligence does not necessarily mean cheap software.

Because the economics are not really about tokens.

They are about what happens after the tokens are generated.

Did the AI solve the problem?

Did it create value?

Did it save time?

Did it reduce risk?

Did it make the right decision?

The future winners of AI will probably not be the companies that simply consume the fewest tokens.

They will be the companies that get the most value from every token they spend.

That is the real token economy.

Not the price of intelligence.

But the return on intelligence.

And in the coming years, that may become one of the most important equations in technology.

Value Created ÷ Intelligence Consumed.

Everything else is just accounting.

Why Sorting Is One of Computing's Deepest Problems

Derek Mwale — Mon, 31 Aug 2026 13:58:56 +0000

There are problems in computer science that look too simple to deserve respect.

Sorting is one of them.

Ask someone what sorting means and the answer sounds almost childish:

Put things in order.

Put the numbers from smallest to largest.

Arrange names alphabetically.

Sort products by price.

Rank players by score.

Order posts by date.

That is it.

At least, that is what sorting looks like from the outside.

But computer science has a strange habit of hiding its deepest ideas inside its simplest-looking problems.

Sorting is not just about rearranging numbers.

Sorting is about information.

It is about what you know and what you do not know.

It is about how efficiently you can reduce uncertainty.

It is about mathematical lower bounds.

It is about memory, hardware, parallelism, distributed systems, databases, and even the limits of algorithms themselves.

Sorting is one of those problems that becomes more interesting the longer you stare at it.

And eventually, you begin to realize something uncomfortable:

Sorting is not really about putting things in order.

Sorting is about discovering order that was hidden inside chaos.

The Problem Looks Trivial Until You Try to Solve It

Suppose I give you these numbers:

7, 2, 9, 1, 5

Your job is to produce:

1, 2, 5, 7, 9

For a human being, this feels almost automatic.

You look at the numbers.

You compare them.

You move them around mentally.

Done.

But a computer does not "see" order.

A computer has operations.

It has memory.

It has instructions.

It has constraints.

It has to physically execute a sequence of steps that transforms:

[7, 2, 9, 1, 5]

into:

[1, 2, 5, 7, 9]

The interesting question is not:

Can we sort the numbers?

Of course we can.

The interesting question is:

How many operations must we perform?

And that single question opens a door into one of the deepest areas of computer science.

Because once we start asking how efficiently something can be sorted, we stop talking about a simple programming exercise.

We start talking about the nature of computation.

Sorting Is a Machine for Destroying Uncertainty

Imagine you have five cards.

Each card contains a number.

They are completely shuffled.

You know nothing about their relative positions.

The number 1 could be anywhere.

The number 9 could be anywhere.

Every possible arrangement is plausible.

For five unique elements, there are:

5! = 120

possible arrangements.

That means before sorting, your uncertainty looks something like this:

120 possible worlds

After sorting, there is only one valid arrangement:

1, 2, 5, 7, 9

Sorting has taken 120 possibilities and reduced them to one.

That is the real nature of the problem.

Every comparison you make gives you information.

For example:

Is 7 > 2?

The answer tells you something about the universe of possible arrangements.

You have eliminated some possibilities.

Then you ask another question:

Is 9 > 1?

More possibilities disappear.

And slowly, through comparisons, you collapse uncertainty.

You are navigating through a massive space of possible orders.

That is why sorting is fundamentally connected to information theory.

The Decision Tree: Where Sorting Becomes Mathematical

Let's imagine a sorting algorithm that can only learn about elements by comparing them.

For example:

a < b?

or:

x > y?

Every comparison has two possible outcomes:

Yes
No

So we can represent the algorithm as a decision tree.

For three elements:

a, b, c

there are:

3! = 6

possible orders.

The algorithm must distinguish between all six.

A comparison tree might look something like this:

                    a < b?
                   /       \
                Yes         No
                /             \
             b < c?          a < c?
             /    \          /     \
          ...      ...     ...     ...

Every comparison splits the universe of possibilities.

But here is the important part.

A binary tree with depth d can have at most:

2^d

leaves.

If we need to distinguish between n! possible permutations, then we need:

2^d >= n!

Taking logarithms:

d >= log₂(n!)

And using mathematical approximations:

log₂(n!) ≈ n log₂(n)

Which gives us one of the most important results in algorithms:

Any comparison-based sorting algorithm requires Ω(n log n) comparisons in the general case.

That is not an implementation problem.

That is not because programmers haven't tried hard enough.

That is not because computers are slow.

It is a mathematical limit.

You Cannot Code Your Way Out of Mathematics

This is one of the most beautiful things about sorting.

For decades, programmers have invented brilliant algorithms.

Merge Sort.

Heap Sort.

Quick Sort.

TimSort.

IntroSort.

Parallel sorting algorithms.

External sorting algorithms.

Cache-aware sorting algorithms.

And yet, for comparison-based sorting, nobody can escape the fundamental lower bound.

You can improve constants.

You can improve memory usage.

You can improve cache locality.

You can improve average performance.

You can exploit special input patterns.

But you cannot simply write a magical comparison sort that says:

I sort arbitrary data in O(n).

Not in the general case.

Because the problem itself contains too much uncertainty.

The algorithm has to discover enough information to distinguish between all possible permutations.

And information has a cost.

This is one of those moments where computer science becomes philosophical.

We often think technology means:

If something is difficult, build a better machine.

But some difficulties are not technological.

Some difficulties are mathematical.

No matter how powerful your computer becomes, some limits remain.

Sorting is one of the places where you can see those limits clearly.

Bubble Sort Is Bad, But It Teaches Something Important

Every programmer has met Bubble Sort.

Usually early.

Usually in school.

Usually before discovering that almost nobody uses it for serious large-scale sorting.

Bubble Sort works by repeatedly swapping neighboring elements.

Something like this:

7, 2, 9, 1, 5

Compare:

7 and 2

Swap:

2, 7, 9, 1, 5

Compare:

7 and 9

No swap.

Then:

9 and 1

Swap:

2, 7, 1, 9, 5

Then:

9 and 5

Swap again.

The largest elements gradually "bubble" toward the end.

The complexity is:

O(n²)

Which becomes painful as n grows.

For a thousand elements:

1,000² = 1,000,000

For a million elements:

1,000,000² = 1,000,000,000,000

A trillion-scale number of operations.

That escalated quickly.

But Bubble Sort is valuable because it teaches us something deeper.

It demonstrates that having a correct algorithm is not enough.

Two algorithms can solve exactly the same problem.

Both can be perfectly correct.

Yet one might finish in milliseconds while the other would make you question your career choices.

This is where programmers first encounter a painful truth:

Correctness and efficiency are completely different achievements.

Merge Sort: The Power of Dividing Chaos

Merge Sort approaches the problem differently.

Instead of trying to sort the entire array at once, it asks:

What if we break the problem into smaller problems?

Take:

[7, 2, 9, 1, 5, 3, 8, 4]

Split it:

[7, 2, 9, 1]     [5, 3, 8, 4]

Split again:

[7, 2] [9, 1] [5, 3] [8, 4]

Again:

[7] [2] [9] [1] [5] [3] [8] [4]

Now something interesting happens.

An array containing one element is already sorted.

So the algorithm begins merging:

[7] + [2] → [2, 7]

Then:

[9] + [1] → [1, 9]

Eventually:

[2, 7] + [1, 9]

becomes:

[1, 2, 7, 9]

The genius of Merge Sort is not the splitting.

The genius is the merging.

Because merging two already sorted lists is cheap.

You simply compare the smallest remaining elements:

[1, 4, 8]
[2, 3, 9]

Compare:

1 vs 2 → take 1

Then:

4 vs 2 → take 2

Then:

4 vs 3 → take 3

And so on.

Every level of merging processes approximately n elements.

And there are approximately:

log₂(n)

levels.

So:

O(n log n)

This is one of the first times many programmers experience the magic of divide and conquer.

Break a difficult problem into smaller pieces.

Solve the smaller pieces.

Combine the results.

It sounds obvious.

But this idea powers an enormous amount of computing.

Quick Sort Is Beautiful Because It Takes Risks

Quick Sort is probably one of the most fascinating sorting algorithms ever created.

Its basic idea is almost reckless.

Choose a value called a pivot.

Then rearrange everything around it.

For example:

[7, 2, 9, 1, 5]

Choose:

pivot = 5

Then divide the elements:

[2, 1]  5  [7, 9]

Everything on the left is smaller.

Everything on the right is larger.

Then recursively sort both sides:

[1, 2]  5  [7, 9]

Done.

The average complexity:

O(n log n)

But the worst case?

O(n²)

Which makes Quick Sort interesting.

It can be extremely fast.

But its performance depends heavily on how well the pivot divides the data.

Choose good pivots and you get beautiful balanced recursion.

Choose terrible pivots and everything falls apart.

This is almost a metaphor for engineering itself.

A small decision near the beginning can determine the shape of everything that follows.

Real Sorting Is Not Just About Big O

This is where computer science education sometimes becomes misleading.

You learn:

Bubble Sort → O(n²)
Merge Sort → O(n log n)
Quick Sort → O(n log n) average

And it feels like the story is finished.

Choose the smallest Big O.

Problem solved.

Except real computers do not execute Big O notation.

Real computers execute instructions.

And instructions interact with hardware.

This means something strange can happen.

An algorithm that looks theoretically better can sometimes be slower in practice.

Why?

Because modern processors care about things like:

CPU caches
memory access patterns
branch prediction
instruction pipelines
memory allocation
recursion overhead

Imagine two algorithms.

Algorithm A performs fewer mathematical operations.

Algorithm B performs slightly more operations but accesses memory sequentially.

On a modern machine, Algorithm B might win.

Because memory is complicated.

The CPU can process data incredibly quickly.

But fetching data from main memory can be expensive compared to accessing data already sitting in cache.

So suddenly sorting becomes a hardware problem.

The algorithm is no longer just competing against mathematics.

It is competing against physics.

The Memory Hierarchy Changes Everything

A computer does not have one giant bucket called "memory."

It has layers.

Something like:

Registers
   ↓
L1 Cache
   ↓
L2 Cache
   ↓
L3 Cache
   ↓
RAM
   ↓
SSD / Disk

Each layer has different properties.

The closer the data is to the CPU, the faster it can usually be accessed.

Now imagine sorting a gigantic dataset.

Millions.

Billions.

Trillions of records.

The dataset might not even fit into RAM.

Now your beautiful in-memory sorting algorithm has a problem.

You cannot simply do:

data.sort();

because data does not fit into memory.

So what do you do?

You enter the world of external sorting.

External Sorting: When Your Data Is Bigger Than Your Computer

Suppose you have:

10 TB of data

Your machine has:

32 GB of RAM

You cannot load everything.

So instead, you process the data in chunks.

For example:

10 TB dataset
      |
      v
Split into chunks
      |
      v
Sort each chunk in memory
      |
      v
Write sorted chunks to disk
      |
      v
Merge the chunks

Visually:

Input Data
    |
    v
+---------+
| Chunk 1 | → Sort → Sorted File 1
+---------+

+---------+
| Chunk 2 | → Sort → Sorted File 2
+---------+

+---------+
| Chunk 3 | → Sort → Sorted File 3
+---------+

            ↓

      Merge Everything

            ↓

      Final Sorted Data

Now the bottleneck is no longer CPU comparisons.

It might be disk I/O.

Reading from storage.

Writing to storage.

Moving data.

At this scale, sorting becomes less about:

Which number is bigger?

and more about:

How do we move information through physical systems efficiently?

This is why sorting is everywhere in databases and distributed computing.

Databases Are Secretly Obsessed With Sorting

Databases love order.

Indexes are about order.

ORDER BY is about order.

Merge joins depend on order.

B-trees maintain ordered structures.

Query planners constantly reason about whether data is already sorted.

Consider:

SELECT *
FROM users
ORDER BY created_at DESC;

Looks innocent.

But imagine:

500 million users

The database has choices.

It could:

Scan everything.
Sort everything.
Return the results.

Or it could use an index.

If the data is already organized around:

created_at

the database might avoid sorting the entire table.

This is the deeper lesson:

The fastest sort is sometimes the one you never have to perform.

Data structure design is often about preserving useful order before someone asks for it.

Sorting and Searching Are Closely Related

A sorted array is more than an organized array.

It becomes a different kind of object.

Consider searching for:

in an unsorted list.

You may need to inspect every element.

Complexity:

O(n)

But in a sorted list, you can use binary search.

Check the middle.

If the target is smaller, ignore half.

If larger, ignore the other half.

Then repeat.

1,000,000 elements

Linear search might inspect close to a million values.

Binary search requires roughly:

log₂(1,000,000) ≈ 20

Twenty comparisons.

That is ridiculous.

Sorting can transform future operations.

You spend computational effort now to create structure.

That structure makes later questions cheaper.

This idea appears everywhere in engineering.

Indexes.

Caches.

Materialized views.

Precomputed data.

Compiled binaries.

Machine learning models.

All of them involve some version of the same tradeoff:

Spend resources now to make future work easier.

The Strange Escape From O(n log n)

Now here is where sorting becomes even more interesting.

Earlier, we said comparison sorting has a lower bound of:

Ω(n log n)

But notice the phrase:

comparison sorting.

That detail matters.

What if we know something about the data?

Suppose we are sorting integers between:

0 and 100

And we have millions of them.

Instead of comparing every number against other numbers, we could count occurrences.

0 → 15 times
1 → 42 times
2 → 7 times
3 → 91 times
...
100 → 22 times

Then output them in order.

This is Counting Sort.

Its complexity can be:

O(n + k)

where k is the range of possible values.

For small ranges, this can approach linear time.

Radix Sort takes this idea further.

Instead of comparing entire numbers, it processes digits.

For example:

Sort by:

ones digit
tens digit
hundreds digit

Eventually the entire list becomes sorted.

And suddenly:

O(n log n)

is no longer the absolute barrier.

Why?

Because the algorithm is using additional information.

This is crucial.

The mathematical lower bound applies when the only thing you can do is compare elements.

Once you exploit the structure of the data, the rules change.

And this is one of the deepest lessons in algorithms:

Constraints are not always limitations. Sometimes they are information.

If you know more about your problem, you can often solve it faster.

Sorting Is Really About Representation

Suppose I give you these objects:

struct User {
    id: u64,
    name: String,
    age: u8,
}

How do you sort them?

By name?

By age?

By ID?

By registration date?

By some calculated reputation score?

The data itself does not contain a single universal order.

Order depends on representation and meaning.

For integers:

2 < 7

seems obvious.

For strings:

"apple" < "banana"

depends on lexicographical rules.

For human names, things become more complicated.

What about:

Åke

What about:

Émile

What about different languages?

What about uppercase and lowercase?

What about Unicode?

What about cultural rules?

Suddenly sorting text is not just a programming problem.

It becomes a linguistic problem.

This is why real-world sorting often requires a comparator:

users.sort_by(|a, b| {
    a.name.cmp(&b.name)
});

But that comparator represents a decision about meaning.

And meaning is often much harder than algorithms.

Stability: When Equal Things Are Not Actually Equal

Imagine these records:

(Alice, Score: 90)
(Bob, Score: 90)
(Chris, Score: 80)

Sort by score.

You might get:

(Alice, 90)
(Bob, 90)
(Chris, 80)

Or:

(Bob, 90)
(Alice, 90)
(Chris, 80)

Both are technically sorted.

But a stable sorting algorithm preserves the original order of equal elements.

Why does this matter?

Imagine first sorting users by name.

Then sorting them by age.

With a stable sort, users of the same age remain alphabetically ordered.

This allows multiple sorting operations to compose.

It is a subtle property.

But subtle properties often become important when systems become complex.

Stability is one of those details that beginners ignore and experienced engineers suddenly care about a lot.

Parallel Sorting: More CPUs Do Not Automatically Make Sorting Easy

You might think:

Sorting is expensive. Why not just use 64 CPU cores?

Good question.

Now the problem becomes:

How do 64 processors coordinate?

Suppose each processor sorts part of the data:

Core 1 → [chunk A]
Core 2 → [chunk B]
Core 3 → [chunk C]
Core 4 → [chunk D]

Great.

But now you have four sorted lists.

You still need to merge them.

And merging introduces communication.

Synchronization.

Memory movement.

Contention.

Load balancing.

This is one of the recurring themes in parallel computing.

The computation itself may be easy.

Coordination is hard.

A hundred processors cannot simply act independently forever.

Eventually they must agree.

And agreement has a cost.

Sorting exposes this beautifully.

Distributed Sorting Is Even More Chaotic

Now imagine the data is not on one machine.

It is spread across:

Server A
Server B
Server C
Server D
Server E

Possibly across different data centers.

Now sorting means moving information across a network.

And networks introduce:

latency
bandwidth limitations
failures
retries
duplicated messages
inconsistent states

Imagine sorting billions of records by a key.

A distributed system might:

1. Partition the data.
2. Send records to different machines.
3. Sort locally.
4. Exchange partitions.
5. Merge results.
6. Recover from failures.

The algorithm now has to survive reality.

A machine might disappear.

A network connection might fail.

One worker might be slower than the others.

One partition might contain far more data than expected.

This is why distributed computing often feels less like writing algorithms and more like negotiating with chaos.

And once again, sorting is sitting right in the middle of it.

Modern Sorting Algorithms Are Hybrids

One of the most interesting things about real-world sorting libraries is that they rarely behave like textbook algorithms.

A standard library sort might combine multiple strategies.

For example:

Small arrays → Insertion Sort
Large partitions → Quick Sort
Bad recursion depth → Heap Sort
Existing runs → Merge strategies

Why?

Because real data is weird.

Sometimes it is already partially sorted.

Sometimes it is nearly sorted.

Sometimes it contains many duplicates.

Sometimes memory is limited.

Sometimes predictable performance matters more than average speed.

The best algorithm depends on reality.

This is why TimSort became famous.

It was designed to exploit patterns that often appear in real-world data.

Real data is frequently not random.

Dates arrive roughly in chronological order.

Logs are often mostly ordered.

Users might already be grouped.

Data might contain long sorted runs.

A smart algorithm can detect structure and exploit it.

And that is another profound lesson:

Random-looking problems are often not truly random.

The world contains patterns.

Good algorithms learn to notice them.

Sorting Is a Lesson in Humility

Sorting begins as one of the first algorithmic problems we encounter.

And yet it keeps returning.

At beginner level:

How do I sort an array?

At intermediate level:

Which algorithm should I use?

At advanced level:

What is the lower bound?

At systems level:

How does cache behavior affect performance?

At database level:

Can we avoid sorting through indexes?

At distributed level:

How do we globally order data across machines?

At theoretical level:

How much information must an algorithm discover?

The problem never really disappears.

It just changes shape.

That is what makes some areas of computer science beautiful.

You can begin with a simple exercise and eventually reach questions about the fundamental limits of computation.

A Small Rust Example

Let's return to something practical.

Rust gives us powerful sorting tools:

fn main() {
    let mut numbers = vec![7, 2, 9, 1, 5];

    numbers.sort();

    println!("{:?}", numbers);
}

Output:

[1, 2, 5, 7, 9]

Simple.

But behind that simplicity is an enormous amount of engineering.

You can also sort by custom logic:

#[derive(Debug)]
struct Player {
    name: String,
    score: u32,
}

fn main() {
    let mut players = vec![
        Player {
            name: "Alice".to_string(),
            score: 90,
        },
        Player {
            name: "Bob".to_string(),
            score: 75,
        },
        Player {
            name: "Chris".to_string(),
            score: 100,
        },
    ];

    players.sort_by(|a, b| b.score.cmp(&a.score));

    for player in players {
        println!("{:?}", player);
    }
}

Now we are sorting by meaning.

Not just numerical position.

We have defined what "before" and "after" mean.

And that is the entire problem of sorting in miniature.

Computers do not naturally understand order.

We define it.

Algorithms discover it.

Machines execute it.

Mathematics limits it.

Hardware complicates it.

And engineering tries to make it fast.

The Deepest Idea: Order Is Information

Perhaps the reason sorting is so important is because ordered information is fundamentally more useful than unordered information.

A pile of documents is data.

A filing system is information architecture.

A list of books is data.

A library catalog is structure.

A database table is data.

An index is order.

A stream of events is chaos.

A timeline is meaning.

Sorting transforms raw collections into structures that can be reasoned about.

Once things are ordered, we can:

search faster
detect duplicates
group similar records
perform efficient joins
find ranges
build indexes
identify patterns
compress data
process streams

Sorting is one of the foundational operations that makes larger computational systems possible.

It is infrastructure disguised as a utility function.

The Universe Loves Disorder. Computers Spend Their Lives Fighting It.

There is something almost poetic about sorting.

You begin with disorder:

8, 3, 1, 7, 2, 9

The computer knows very little.

Every element could belong almost anywhere.

Then, through comparisons, partitions, merges, counts, and memory operations, the machine slowly creates certainty.

Eventually:

1, 2, 3, 7, 8, 9

Chaos has become structure.

And perhaps that is why sorting keeps appearing everywhere in computing.

Computers are constantly doing this.

They organize.

Index.

Rank.

Group.

Prioritize.

Schedule.

Search.

Classify.

Compress.

Compute.

Behind many of those operations is the same fundamental idea:

Find structure inside a collection of possibilities.

Sorting is one of the earliest examples of that idea.

And one of the deepest.

Final Thoughts

Sorting looks like a solved problem.

In one sense, it is.

We have excellent algorithms.

Modern programming languages provide highly optimized implementations.

You can sort a list with a single line of code.

But solved problems can still be profound.

Sorting teaches us about:

algorithmic complexity
mathematical lower bounds
information theory
divide and conquer
recursion
memory hierarchies
hardware architecture
databases
distributed systems
parallel computing
data representation

Few programming problems have such a small surface area and such enormous depth.

That is what makes sorting special.

It begins with:

Put these things in order.

And ends with:

What is the minimum amount of information required to transform uncertainty into certainty?

That is no longer just a question about arrays.

That is a question about computation itself.

And maybe that is the strangest thing about computer science.

Sometimes the deepest problems are hiding inside the functions we stop thinking about.

numbers.sort();

One line.

Thousands of years of mathematics behind the idea of order.

Decades of computer science behind efficient algorithms.

Billions of transistors executing instructions.

Caches moving data.

Branches being predicted.

Memory being allocated.

Processors coordinating.

And somewhere underneath all of it, the same ancient human instinct:

Take the chaos.

Understand it.

Put it in order.

That is sorting.

And that is why one of the simplest problems in programming is also one of the deepest.

Why Code Reviews Feel Personal

Derek Mwale — Mon, 31 Aug 2026 13:52:59 +0000

The pull request is about the code. So why does it sometimes feel like someone just criticized your intelligence?

There is a strange moment in every programmer’s life.

You spend three hours solving a problem.

Maybe five.

Maybe the entire weekend.

You stare at a bug that refuses to explain itself. You rewrite the same function four times. You search documentation. You question your career. You finally discover that the problem was a missing null check, a race condition, an incorrect database query, or a variable that was initialized in the wrong place.

Eventually, it works.

The tests pass.

The application stops crashing.

You push your branch.

You open a pull request.

And then someone leaves this comment:

“Could we simplify this?”

Suddenly, your heart rate increases.

Another comment appears.

“I think this should probably be moved into a separate service.”

Then another.

“Why are we doing this here?”

And now, somehow, a conversation about a function with 27 lines has become an existential crisis.

You read the comments again.

Maybe the reviewer is right.

Maybe the code really is complicated.

Maybe the architecture is inconsistent.

But that is not what your brain hears.

Your brain hears:

“You are not good enough.”

That is why code reviews are strange.

Technically, they are supposed to be simple.

Someone writes code.

Someone else reads it.

The team improves it.

Everyone ships better software.

But humans are involved.

And humans have egos, identities, insecurities, histories, ambitions, and an almost supernatural ability to interpret a comment about a for loop as a personal attack.

The code review is not personal.

But it often feels personal.

And understanding why might be one of the most important things you can learn as a programmer.

Code Is More Than Code

To understand why reviews feel personal, we first need to understand something uncomfortable:

Programmers do not just write code. They put themselves into it.

Every piece of code contains decisions.

Why did you use this algorithm?

Why did you choose PostgreSQL instead of MongoDB?

Why is this function asynchronous?

Why is this abstraction here?

Why did you create three classes instead of one?

Why did you use Rust?

Why is there a microservice for something that could probably be a function?

These decisions are not random.

They come from your experience.

Your education.

The tutorials you watched.

The bugs you survived.

The codebases that traumatized you.

The senior developer who once told you that global state was evil.

The startup where everything was a microservice.

The company where everything was one giant PHP file.

Your code is a compressed representation of your engineering history.

That means when someone says:

“I wouldn't do it this way.”

They may only be talking about the implementation.

But because the implementation came from you, it is easy to hear:

“I wouldn't think the way you think.”

That is the first problem.

We confuse criticism of the artifact with criticism of the creator.

A painting is criticized.

The painter feels criticized.

An article is edited.

The writer feels criticized.

A song is rejected.

The musician feels rejected.

And in software engineering, a function gets reviewed.

The programmer quietly starts questioning whether they should have studied accounting instead.

The Pull Request Is a Public Performance

Writing code privately feels very different from having your code reviewed.

When you are coding alone, you are in control.

You can experiment.

You can write terrible temporary code.

You can create a variable called:

let thing2 = thing1;

Nobody knows.

You can promise yourself you will clean it later.

Maybe you will.

Maybe you won't.

That is between you and your Git history.

But the moment you open a pull request, something changes.

Your work becomes visible.

Other engineers can see your decisions.

They can see what you missed.

They can see what you misunderstood.

They can see that the elegant architecture you described in the stand-up meeting somehow resulted in a 400-line function called:

process_everything()

And now you are exposed.

This is why code reviews can feel less like collaboration and more like an examination.

You are no longer simply writing software.

You are presenting your thinking.

Every line becomes evidence.

Every abstraction becomes a statement.

Every missing test becomes suspicious.

Every comment from a reviewer feels like a red pen on an exam paper.

The pull request becomes a stage.

And suddenly, engineering becomes performance.

We Attach Intelligence to Programming

Programming has another problem.

Many programmers connect their identity to intelligence.

This is dangerous.

From the moment people start learning programming, they hear stories about geniuses.

The teenager who built a billion-dollar company.

The legendary engineer who writes operating systems.

The programmer who understands distributed systems before breakfast.

The person who contributes to the Linux kernel while everyone else is still figuring out CSS.

Technology has created a strange mythology around intelligence.

We talk about “10x engineers.”

We celebrate people who can solve impossible algorithm problems.

We create competitive programming contests.

We measure people with technical interviews that sometimes feel like mathematical gladiator arenas.

Eventually, many developers internalize a simple idea:

If I am good at programming, I am intelligent.

Unfortunately, the reverse also becomes true.

If my code is bad, maybe I am not intelligent.

And this is where code reviews become emotionally dangerous.

Because a reviewer saying:

“This has a bug.”

Can accidentally become:

“You are not as smart as you think you are.”

A reviewer saying:

“This could be cleaner.”

Can become:

“Your thinking is messy.”

A reviewer suggesting a simpler solution can become:

“You made this unnecessarily complicated because you don't understand the problem.”

None of those things may actually be true.

But our brains are excellent at constructing emotional fan fiction.

The Effort Problem

There is another reason code reviews hurt.

Reviewers usually see the code. They do not see the struggle.

Imagine this.

You spent eight hours solving a problem.

You tried six approaches.

Three failed.

One caused production errors.

One worked but was too slow.

One created a memory leak.

Eventually, after hours of debugging, you arrive at the solution.

The reviewer opens the pull request.

They spend five minutes reading it.

Then they write:

“I think this can be done in one query.”

You stare at the comment.

One query?

ONE QUERY?

Where was this person when you were suffering?

Where were they when you had twelve browser tabs open?

Where were they when the debugger looked like a crime scene?

Where were they when you almost deleted the repository and started farming?

The reviewer sees the final result.

They do not see the journey.

And humans naturally value effort.

If something took us a long time, we unconsciously believe it deserves respect.

So when someone casually suggests replacing our eight-hour solution with something simpler, it can feel disrespectful.

Not because they are wrong.

But because they accidentally ignored the invisible work behind the code.

This is one of the most important psychological realities in software engineering:

The amount of effort required to discover a solution is often unrelated to how complicated the final solution looks.

Great solutions frequently look obvious after someone discovers them.

That does not mean they were obvious before.

Code Reviews Trigger Ownership

When you write code, something strange happens.

You begin to own it.

Even though the repository belongs to the company.

Even though the application belongs to the users.

Even though the code may be deleted next week.

You still think:

My code.

And ownership creates attachment.

Imagine you build this:

pub fn calculate_total(items: &[Item]) -> f64 {
    items.iter().map(|item| item.price).sum()
}

Someone comments:

“Can we rename this to sum_prices?”

Technically, this is a tiny suggestion.

Emotionally?

You may suddenly become a constitutional lawyer defending the original function name.

You explain the semantic meaning.

You discuss future extensibility.

You mention domain-driven design.

You cite three software architecture books.

You create a 14-message thread.

All because someone wanted to rename a function.

Why?

Because the discussion is no longer about the function name.

It is about ownership.

When someone changes something we created, our brains sometimes interpret it as someone entering our territory.

This is irrational.

But it is human.

The healthiest engineers eventually learn something extremely liberating:

The code is not you.

You wrote it.

That does not mean you have to defend it forever.

You can delete it.

Refactor it.

Replace it.

Rewrite it.

Your identity survives.

The compiler does not care about your feelings.

And honestly, neither does production.

The Architecture of Ego

Ego in software engineering is fascinating because it often hides behind technical language.

Nobody says:

“I feel personally attached to this abstraction.”

Instead, they say:

“This architecture is more scalable.”

Nobody says:

“I don't want to admit my approach was wrong.”

Instead, they say:

“There are trade-offs.”

Nobody says:

“Changing this would hurt my pride.”

Instead, they say:

“We should avoid unnecessary refactoring.”

And sometimes those statements are true.

There really are trade-offs.

Architecture really matters.

Refactoring really can introduce risk.

But sometimes technical arguments are simply ego wearing a hoodie.

That is one of the hardest things to recognize in yourself.

Because ego rarely introduces itself.

It does not knock on the door and say:

“Hello, Derek. You are defending this terrible implementation because you don't want to admit Sarah found a better solution.”

No.

It gives you a much more sophisticated argument.

It tells you:

“The existing implementation is optimized for future scalability requirements.”

Meanwhile, there are six users.

Senior Developers Can Accidentally Make Reviews Worse

Experience creates another strange dynamic.

Senior developers often see problems faster.

That is their advantage.

They have seen similar bugs.

Similar architectures.

Similar failures.

They can look at something and immediately think:

Oh no. This is going to hurt later.

The problem is that expertise can create impatience.

A senior developer sees an obvious issue.

The junior developer spent two days solving it.

The senior developer writes:

“This is wrong.”

Technically accurate.

Emotionally catastrophic.

Because communication matters.

There is a massive difference between:

“This is wrong.”

And:

“I think this approach might create a problem when we have concurrent requests. What do you think about handling the state differently?”

Both comments may lead to the same technical solution.

But one treats the author like a defective machine.

The other treats them like an engineer.

Good code reviews are not just about detecting mistakes.

They are about transferring understanding.

The best reviewer does not simply say:

“Fix this.”

They help answer:

Why is this a problem?

Because when someone understands the reason, they become better.

When they only follow instructions, they become dependent.

The Difference Between a Review and a Judgment

This distinction is everything.

A review examines the work.

A judgment examines the person.

A review says:

“This function has too many responsibilities.”

A judgment says:

“You don't understand clean architecture.”

A review says:

“We might have a race condition here.”

A judgment says:

“You clearly don't understand concurrency.”

A review says:

“Could we add a test for this edge case?”

A judgment says:

“Why do you never write tests?”

The technical content may be similar.

But the psychological effect is completely different.

The first focuses on the software.

The second assigns identity.

And once identity enters the conversation, people become defensive.

Because now the discussion is no longer:

How do we improve this system?

It becomes:

Am I a good programmer?

That is a much harder question to answer in a GitHub comment.

Why Tone Is So Difficult in Text

Code reviews happen mostly through text.

And text is terrible at carrying emotional nuance.

Consider this comment:

“Why did you do this?”

Maybe the reviewer is genuinely curious.

Maybe they want context.

Maybe they are trying to understand a business requirement.

But the author may read it as:

“What kind of idiot would do this?”

The words are identical.

The tone exists entirely inside the reader's head.

That makes asynchronous communication dangerous.

You cannot see facial expressions.

You cannot hear hesitation.

You cannot tell whether someone is smiling.

You cannot hear:

“Why did you do this? 🙂”

Instead, you see:

WHY DID YOU DO THIS.

And suddenly you are preparing for war.

This is why experienced teams often develop review cultures around phrases like:

“What do you think about…”
“Could we consider…”
“I'm wondering if…”
“Would this cause…”
“Maybe we could…”
“Non-blocking suggestion:”

These phrases are not just corporate politeness.

They reduce unnecessary emotional friction.

The Dangerous Phrase: “Obviously”

There are few words more destructive in software engineering than:

“Obviously.”

Nothing is obvious to everyone.

If something is obvious to you, it usually means you have information the other person does not.

Experience makes patterns obvious.

Context makes decisions obvious.

History makes architecture obvious.

But saying:

“Obviously this should be asynchronous.”

Does not teach anyone anything.

It only communicates:

“I understand something you don't.”

That creates hierarchy.

And hierarchy creates insecurity.

A much better comment is:

“If this runs synchronously, could it block the request thread under heavy load? Would moving it to a background job help?”

Now the reviewer is still demonstrating expertise.

But they are also sharing the reasoning.

That changes everything.

Junior Developers Often Feel Reviews More Deeply

When you are experienced, you have evidence that you can survive being wrong.

You have written terrible code before.

You have broken production.

You have fixed impossible bugs.

You have rewritten entire systems.

You know that being wrong is not the end of your career.

But junior developers do not have that emotional history yet.

Every mistake feels significant.

Every rejected pull request feels like evidence.

Every comment feels like someone adding another entry to a secret list titled:

Reasons We Should Never Have Hired This Person.

Of course, that list probably does not exist.

But anxiety does not require evidence.

That is why mentorship matters.

A good senior engineer understands that the goal is not simply to produce perfect code today.

The goal is to help produce stronger engineers tomorrow.

Sometimes that means being technically correct and emotionally intelligent.

Code Review Culture Can Make or Break a Team

Two companies can use the exact same technology stack.

The same programming language.

The same CI pipeline.

The same repository structure.

The same cloud provider.

And have completely different engineering cultures.

At Company A:

Every pull request feels like a courtroom.

People defend every decision.

Reviewers hunt for mistakes.

Comments are aggressive.

Nobody wants to look stupid.

Developers avoid asking questions.

Junior engineers copy patterns without understanding them.

People optimize for being right.

At Company B:

Reviews are conversations.

People ask questions.

Reviewers explain trade-offs.

Authors are comfortable saying:

“You're right. I missed that.”

Senior developers admit mistakes.

Junior developers challenge assumptions.

People optimize for learning.

Technically, both companies review code.

Culturally, they are doing completely different things.

One is using code review as a gatekeeping mechanism.

The other is using it as a knowledge-sharing system.

The difference is enormous.

The Best Engineers Are Comfortable Being Wrong

This might be one of the most underrated engineering skills.

Not intelligence.

Not speed.

Not knowing every framework.

The ability to say:

“You're right.”

That is powerful.

Imagine discovering that your entire approach was wrong.

Your database schema.

Your abstraction.

Your API design.

Your concurrency model.

Someone points out a better solution.

You have two choices.

Option One

Defend everything.

Find reasons why the new approach is risky.

Bring up edge cases.

Mention theoretical scalability.

Create increasingly complicated arguments.

Eventually win the discussion.

And keep worse code.

Option Two

Say:

“Good catch. I didn't consider that.”

Fix it.

Move on.

The second option requires confidence.

Insecure people often think admitting mistakes makes them look weak.

In reality, the opposite is usually true.

Strong engineers do not need to pretend they are always correct.

They know they will be wrong again.

And they are comfortable with that.

Your First Draft Is Not Your Final Identity

One reason programmers struggle with reviews is because they treat the first implementation as a declaration.

They think:

This is my solution.

But software engineering is not about producing perfect thoughts on the first attempt.

It is iterative.

The first version is supposed to be imperfect.

That is why we have:

Pull requests
Tests
Staging environments
CI pipelines
Refactoring
Monitoring
Bug reports
Version control

The entire software industry is basically a giant admission that humans do not get things right immediately.

If every programmer could produce perfect software alone, Git would be unnecessary.

Teams would be unnecessary.

Code reviews would be unnecessary.

Production would probably still find a way to crash, but at least we would feel confident.

The review exists because your first idea is not necessarily your best idea.

And that is okay.

How to Receive Code Reviews Without Losing Your Mind

Here are a few principles that have helped me think about reviews differently.

1. Assume the Reviewer Is Attacking the Code, Not You

Unless someone is clearly being disrespectful, start with the most charitable interpretation.

They found a problem.

That is literally part of the process.

If someone says:

“This could cause a memory leak.”

Do not immediately start analyzing their personality.

Check whether it could cause a memory leak.

The bug does not care who wins the argument.

2. Ask for the Reason

If you disagree with a suggestion, do not immediately defend yourself.

Ask:

“Can you explain the concern here?”

Sometimes you will discover that the reviewer has context you don't.

Sometimes they will discover that you had context they didn't.

Either way, the conversation becomes technical rather than emotional.

3. Don't Defend Every Line

You do not need to be the lawyer for every variable you have ever named.

Sometimes a reviewer suggests something better.

Accept it.

You are allowed to change your mind.

The world will continue spinning.

4. Separate Preference From Correctness

This is a major source of pointless arguments.

Some things are objectively dangerous.

A SQL injection vulnerability is not a style preference.

A race condition may not be a style preference.

A missing authentication check is definitely not a style preference.

But:

Tabs versus spaces?

Function naming?

File organization?

Brace placement?

Sometimes engineers fight as if these questions determine the future of civilization.

They don't.

Use linters.

Use formatters.

Automate the boring arguments.

Save your emotional energy for problems that matter.

How to Give Better Code Reviews

Being a good reviewer is also a skill.

Here are a few principles.

Review the Code, Not the Person

Bad:

“You don't understand how this works.”

Better:

“I think this might behave unexpectedly when multiple requests arrive simultaneously.”

The second one teaches.

The first one humiliates.

Explain the Why

Instead of:

“Change this.”

Try:

“Could we change this? I'm concerned that the current version might create duplicate records if the request is retried.”

Now the author learns something.

Separate Blocking Issues From Suggestions

Not everything needs to stop the pull request.

Use categories mentally:

Blocking

This must be fixed.

Important

This should probably be fixed.

Suggestion

This might improve readability.

Nitpick

I personally prefer this.

Without this distinction, developers can feel like every comment is equally serious.

And suddenly:

“Rename this variable.”

Feels as urgent as:

“This endpoint exposes every user's password.”

They are not the same problem.

Hopefully.

Leave Positive Feedback Too

Code review should not become a bug detector.

Sometimes someone makes a good decision.

Say it.

“Nice handling of the retry logic here.”

“Good test coverage.”

“I like this abstraction.”

“This is much simpler than the previous implementation.”

Positive feedback helps reinforce good engineering decisions.

It also reminds developers that reviews are not just a list of reasons they are disappointing.

The Real Goal of Code Review

The ultimate goal is not perfect code.

There is no such thing.

Every system contains compromises.

Every architecture contains assumptions.

Every abstraction eventually annoys someone.

The goal of code review is to improve the probability that the software survives reality.

Reality is brutal.

Users will enter unexpected data.

Networks will fail.

Servers will restart.

Databases will slow down.

APIs will return nonsense.

Someone will deploy on Friday.

Someone will accidentally remove an environment variable.

Someone will discover that the “temporary solution” has been running in production for four years.

Code review is a defense mechanism against human limitations.

You are not reviewing code because the author is incompetent.

You are reviewing it because everyone is blind to some of their own mistakes.

That includes senior engineers.

Especially senior engineers.

Experience does not make you incapable of mistakes.

It just gives you more sophisticated ways to make them.

The Humbling Truth

Here is the truth every programmer eventually learns.

Some of the code you are incredibly proud of today will embarrass you in five years.

Maybe sooner.

You will open an old repository and think:

Who wrote this?

Then you will check Git.

And discover:

Oh. Me.

This is actually good news.

If your old code embarrasses you, it probably means you improved.

The goal is not to create code you will defend forever.

The goal is to become the kind of engineer who can look at old decisions and say:

“That made sense with what I knew then. I know more now.”

That is growth.

And code reviews accelerate that growth.

They expose you to other ways of thinking.

Other patterns.

Other experiences.

Other mistakes.

A good reviewer can save you from discovering something the hard way.

And the hard way is usually production.

Conclusion: It's Personal Because We Are Human

Code reviews feel personal because programming is deeply connected to identity.

We spend hours thinking.

Designing.

Debugging.

Building.

We make decisions with incomplete information.

Then we expose those decisions to other people.

Of course it feels vulnerable.

But vulnerability is not weakness.

It is part of collaboration.

The healthiest engineering teams understand something important:

The person is not the code.

The code can be wrong while the programmer is talented.

The code can be messy while the engineer is learning.

The architecture can fail while the decision was reasonable.

A pull request can be rejected without rejecting the person who wrote it.

And sometimes the best thing someone can do for your growth is point at something you spent three days building and say:

“Have you considered another approach?”

That comment might annoy you.

You might stare at it for five minutes.

You might temporarily question the reviewer's ancestry.

But eventually, if the team culture is healthy, you will realize something.

They were not trying to destroy your work.

They were trying to improve it.

And maybe improve you along the way.

The best engineers are not the ones whose code never gets criticized.

Those engineers probably aren't letting anyone look at their code.

The best engineers are the ones who can survive the uncomfortable moment when someone says:

“I think we can do better.”

Then look at the code.

Take a breath.

And answer:

“Show me.”

Because in the end, that is what code review should be.

Not a battle.

Not an exam.

Not a public trial of your intelligence.

Just a group of imperfect humans trying to make fewer mistakes before the software reaches millions of other imperfect humans.

And honestly?

That might be the most human thing about programming.

The code may be logical.

The machines may be precise.

The compiler may be brutally honest.

But the people writing the software?

We are emotional.

We are insecure.

We are proud.

We are curious.

We are wrong.

We learn.

And somewhere between a pull request, a red comment, and a slightly passive-aggressive suggestion about renaming a variable, we slowly become better engineers.

Maybe that is why code reviews feel personal.

Because behind every line of code is a person saying:

“This is how I thought the problem should be solved.”

And behind every good review is another person saying:

“Interesting. Let's think about it together.”

The Mathematics Behind Database Performance

Derek Mwale — Sat, 29 Aug 2026 09:22:37 +0000

Databases Don't Become Slow by Accident. There Is Usually an Equation Hiding Somewhere.

There is a moment every backend developer eventually experiences.

The application works.

The API works.

The frontend looks beautiful.

The database has a few thousand rows.

Everything feels ridiculously fast.

Then the application grows.

A few thousand rows become a few million.

A few hundred users become tens of thousands.

The innocent query that used to take:

2 ms

now takes:

2 seconds

Then:

8 seconds

Then someone opens the dashboard and suddenly the entire backend begins behaving like it is reconsidering its life choices.

Developers start doing what developers traditionally do.

They add an index.

Sometimes it works.

Sometimes it doesn't.

Then they add another index.

Then another.

Then someone says:

"Let's increase the database server."

So the CPU goes from 4 cores to 16.

RAM goes from 16 GB to 64 GB.

The database becomes faster.

For a while.

Then the dataset grows again.

And eventually you discover something uncomfortable:

Database performance is not magic.

It is mathematics.

Underneath every database query there are relationships involving:

latency
throughput
probability
logarithms
memory
storage bandwidth
queueing
cache hit rates
selectivity
cardinality
network round trips
concurrency
asymptotic complexity

A database is a mathematical machine pretending to be a table.

And understanding the mathematics behind it changes how you design systems.

Because suddenly an index is not just "something that makes queries faster."

It becomes a data structure with a mathematical cost.

A join is not just "combining tables."

It becomes an algorithm whose complexity depends on the sizes and distributions of its inputs.

A cache is not just "fast memory."

It becomes a probability distribution.

A database server is not just a computer.

It becomes a queue.

And a query is not just SQL.

It is a computational problem.

Let's go down the rabbit hole.

1. The First Equation: Time

Let's start with the simplest possible model.

Suppose a query takes:

T = 50 ms

If one request runs at a time, you can theoretically execute:

1000 ms / 50 ms = 20

queries per second.

So:

$$
Throughput \approx \frac{1}{Latency}
$$

If a query takes 100 ms:

$$
\frac{1}{0.1} = 10
$$

queries per second per sequential execution path.

If it takes 1 ms:

$$
\frac{1}{0.001} = 1000
$$

The first important lesson is obvious:

Lower latency generally means higher potential throughput.

But real databases are not single-threaded calculators.

They have concurrency.

They have queues.

They have locks.

They have disks.

They have network connections.

They have CPU contention.

They have cache misses.

So a more useful approximation is:

$$
Throughput \approx \frac{Concurrency}{Latency}
$$

Suppose you have 100 concurrent workers and average request latency is 50 ms:

$$
\frac{100}{0.05} = 2000
$$

requests per second.

But if latency increases to 500 ms:

$$
\frac{100}{0.5} = 200
$$

requests per second.

Same concurrency.

Same application.

Same hardware.

Ten times slower latency.

Potential throughput drops roughly tenfold.

This is why latency is not merely a user-experience metric.

It is an architectural constraint.

2. Little's Law Is Hiding Inside Your Backend

One of the most useful equations in performance engineering comes from queueing theory.

It is known as Little's Law:

$$
L = \lambda W
$$

Where:

(L) = average number of items in the system
(\lambda) = arrival rate
(W) = average time spent in the system

For databases, you can interpret this as:

$$
Concurrency = Throughput \times Latency
$$

Imagine your API receives:

1000 requests/sec

and each request spends:

50 ms

waiting on the database.

Then:

$$
L = 1000 \times 0.05
$$

$$
L = 50
$$

Approximately 50 requests are simultaneously in the database portion of the system.

Now suppose a slow query increases latency to:

500 ms

At the same arrival rate:

$$
L = 1000 \times 0.5
$$

$$
L = 500
$$

Now your system needs to sustain roughly 500 concurrent database operations.

That means connection pools grow.

Queues grow.

Memory usage grows.

Lock contention grows.

Context switching grows.

The original query being "only 10x slower" can create much more than a 10x operational problem.

This is one reason latency spikes can become cascading failures.

3. Why Indexes Work: Logarithms

Now we reach the most famous mathematical idea behind database indexing.

Suppose your table contains:

1,000,000 rows

You need to find one row.

A naive scan might inspect:

$$
O(n)
$$

rows.

In the worst case:

1,000,000 comparisons

That is linear search.

But a balanced tree index changes the problem.

Instead of asking:

"Which row is this?"

you repeatedly divide the search space.

This gives approximately:

$$
O(\log n)
$$

complexity.

For one million entries:

$$
\log_2(1,000,000) \approx 20
$$

So instead of potentially examining a million entries, a tree may need around twenty levels of decisions.

That is an absurd difference.

Consider:

Linear:
1,000,000

Binary/tree-like:
~20

This is why indexes feel magical.

But they are not magic.

They are exploiting logarithmic mathematics.

4. B-Trees and Why Databases Love Them

Most traditional relational databases rely heavily on B-tree-family indexes.

Why?

Because databases do not live entirely in RAM.

Storage access matters.

Suppose each index node can contain hundreds of keys.

Then the branching factor can be enormous.

Imagine a simplified tree where each node has:

100 children

Then:

$$
100^1 = 100
$$

$$
100^2 = 10,000
$$

$$
100^3 = 1,000,000
$$

$$
100^4 = 100,000,000
$$

Four levels can theoretically cover an enormous number of records.

This is one of the beautiful things about high branching factors.

The database does not need a 20-level tree.

It might need only a handful of page accesses.

And that matters because database performance is often dominated not by CPU instructions but by memory and storage behavior.

5. The Real Enemy: I/O

Modern CPUs are ridiculously fast.

Storage is much slower.

Memory sits somewhere in between.

This creates a hierarchy:

CPU registers
      ↓
CPU cache
      ↓
RAM
      ↓
SSD
      ↓
Network storage

Each layer has different latency characteristics.

This means:

Database performance is frequently a problem of moving bytes.

Not calculating numbers.

Moving bytes.

Suppose a query can be answered from memory.

Excellent.

Suppose the database must read thousands of disk pages.

Now things become interesting.

Imagine:

1 page = 8 KB

and the query needs:

10,000 pages

Then:

$$
10,000 \times 8KB = 80MB
$$

The database potentially needs to move 80 MB of data.

Even if the CPU can process it quickly, the storage subsystem and memory hierarchy still have to move it.

This is why reducing the amount of data touched often matters more than optimizing the number of CPU instructions.

6. Selectivity: The Mathematics of "How Much?"

Suppose you have:

10,000,000 users

and run:

SELECT *
FROM users
WHERE country = 'Zambia';

If 1% of users are in Zambia:

$$
Selectivity = 0.01
$$

Expected rows:

$$
10,000,000 \times 0.01 = 100,000
$$

That's still a lot.

Now imagine:

WHERE user_id = 123456

If user_id is unique:

$$
Selectivity \approx \frac{1}{10,000,000}
$$

Expected result:

1 row

This distinction matters because an index is useful when it significantly reduces the amount of data that must be examined.

The database optimizer is constantly asking a question that sounds almost philosophical:

"How much of the table am I going to have to touch?"

If the answer is:

0.00001%

an index is extremely attractive.

If the answer is:

70%

the optimizer might decide scanning the table is actually cheaper.

And this is why blindly adding indexes is not database optimization.

The optimizer is solving a cost problem.

7. Why Sometimes a Full Table Scan Wins

This confuses many developers.

You create an index.

You expect the database to use it.

But the query planner chooses:

Seq Scan

You think:

"The database is stupid."

Usually, it isn't.

Suppose a table has:

1,000,000 rows

and your query wants:

700,000 rows

Using an index might involve:

Walking the index.
Finding hundreds of thousands of references.
Fetching rows from the table.
Performing many random accesses.

A sequential scan may simply read the table efficiently from beginning to end.

If storage can stream data efficiently, the sequential approach can win.

Mathematically, you can think of the optimizer as comparing:

$$
Cost_{index}
$$

against:

$$
Cost_{scan}
$$

and selecting approximately:

$$
min(Cost_{index}, Cost_{scan})
$$

That is database optimization in miniature.

8. Composite Indexes Are Geometry

Now consider:

SELECT *
FROM orders
WHERE customer_id = 42
AND created_at > '2026-01-01';

You might create:

CREATE INDEX idx_orders_customer_date
ON orders(customer_id, created_at);

Why this order?

Because a composite index creates an ordering over multiple dimensions.

Conceptually:

(customer_id, created_at)

means the index is primarily ordered by:

customer_id

and then within each customer:

created_at

Think of it as sorting a spreadsheet first by column A and then by column B.

This means:

(customer_id, created_at)

is not equivalent to:

(created_at, customer_id)

The mathematics of ordering matters.

If the query primarily narrows by customer_id, putting it first can allow the database to jump into a much smaller region of the index.

This is why index design is not:

"Index all the columns."

It is:

"Design an ordering that makes common queries cheap."

9. The Leftmost Prefix Idea

Suppose we have:

INDEX(a, b, c)

The index naturally supports access patterns beginning with:

or:

a, b

or:

a, b, c

But a query using only:

does not necessarily get the same benefit.

Why?

Because the index is ordered like:

a → b → c

not:

b → a → c

This is another example of mathematics hiding behind SQL.

An index is an ordering.

An ordering determines which search problems can be solved efficiently.

Database engineers are basically designing multidimensional search structures while pretending they are just writing SQL.

10. Join Algorithms Are Algorithms

Consider two tables:

users
orders

Suppose:

users = 1,000,000 rows
orders = 10,000,000 rows

A join is not just a SQL concept.

It is an algorithmic problem.

The database has several strategies.

One is nested loop join.

Conceptually:

for each user:
    find matching orders

In naive form:

$$
O(nm)
$$

If:

$$
n = 1,000,000
$$

and:

$$
m = 10,000,000
$$

then:

$$
n \times m = 10^{13}
$$

That is:

10,000,000,000,000

potential comparisons.

Absolutely terrible.

But an index changes the equation.

For each user, the database can search orders through an index in roughly:

$$
O(\log m)
$$

So the conceptual cost becomes:

$$
O(n\log m)
$$

which is dramatically smaller.

This is why indexes on join keys can be transformative.

11. Hash Joins: Trading Memory for Speed

Another approach is a hash join.

Suppose you build a hash table for one relation.

Conceptually:

Build hash table
      ↓
Scan second table
      ↓
Lookup matching keys

Hash lookup is approximately:

$$
O(1)
$$

on average.

So instead of repeatedly searching the entire relation, you perform constant-time-ish lookups.

The overall cost can approach:

$$
O(n+m)
$$

assuming reasonable hashing behavior and enough memory.

This is beautiful because it demonstrates one of the central ideas in computer science:

Spend more memory to save time.

A hash table stores extra information so that future searches become cheap.

Indexes do something similar.

Caches do something similar.

Materialized views do something similar.

Denormalization does something similar.

A huge percentage of performance engineering is really:

"Can I spend space now to avoid work later?"

12. Sorting Is More Expensive Than People Think

Suppose a query asks:

ORDER BY created_at

Sorting (n) records generally costs approximately:

$$
O(n\log n)
$$

For:

1,000,000 rows

you are dealing with roughly:

$$
1,000,000 \times \log_2(1,000,000)
$$

which is approximately:

$$
20,000,000
$$

units of comparison-scale work.

But if you already have an index ordered by created_at, the database may be able to retrieve records in sorted order without performing a full sort.

This is another reason indexes are more than lookup accelerators.

They can encode ordering.

And ordering can eliminate computation.

13. Pagination Is a Complexity Problem

Everyone starts with:

SELECT *
FROM posts
ORDER BY id
LIMIT 20
OFFSET 1000000;

It looks harmless.

It is not always harmless.

As the offset increases, the database may need to walk past a large number of records before returning the requested page.

Conceptually:

$$
Cost \approx O(offset + limit)
$$

So:

OFFSET 20

is cheap.

But:

OFFSET 1,000,000

can be expensive.

This is why cursor-based pagination is so powerful.

Instead of saying:

"Skip one million records."

you say:

"Start after this known key."

For example:

SELECT *
FROM posts
WHERE id > 1000000
ORDER BY id
LIMIT 20;

With an appropriate index, the database can jump directly into the relevant region.

The cost becomes closer to:

$$
O(\log n + k)
$$

where (k) is the number of returned records.

This is a beautiful example of algorithmic thinking improving API design.

Pagination is not just a frontend concern.

It is a database complexity concern.

14. The N+1 Query Problem Is Multiplication

Suppose your API returns:

100 posts

Then for each post, it fetches its author.

You now have:

1 query for posts
+
100 queries for authors

Total:

$$
101
$$

queries.

If there are:

10,000 requests

per minute:

$$
101 \times 10,000
$$

equals:

$$
1,010,000
$$

database queries per minute.

That is:

$$
16,833
$$

queries per second.

And suddenly the problem becomes obvious.

The database was not necessarily slow.

The application multiplied work.

The N+1 problem is essentially a multiplication problem hiding inside application logic.

15. Caching Is Probability

Now let's talk about caching.

Suppose a database query normally costs:

20 ms

But a cache lookup costs:

0.5 ms

If the cache hit rate is:

95%

then:

$$
P(hit)=0.95
$$

and:

$$
P(miss)=0.05
$$

Expected latency is approximately:

$$
E[T] = P(hit)T_{hit} + P(miss)T_{miss}
$$

Therefore:

$$
E[T] = 0.95(0.5) + 0.05(20)
$$

$$
E[T] = 0.475 + 1
$$

$$
E[T] = 1.475ms
$$

Your average latency dropped from:

20 ms

to approximately:

1.475 ms

because of probability.

But there is another lesson.

A cache hit rate of:

95%

sounds excellent.

But if the cache miss path is extremely expensive, those 5% misses can dominate tail latency.

This is why average latency is not enough.

16. The Mathematics of Tail Latency

Suppose:

99% of requests = 10 ms
1% of requests = 2 seconds

Average latency:

$$
0.99(10) + 0.01(2000)
$$

$$
= 9.9 + 20
$$

$$
= 29.9ms
$$

You might report:

"Our average latency is about 30 ms."

Sounds excellent.

Until 1% of your users experience 2 seconds.

At millions of requests, that 1% is not a corner case.

If you process:

10,000,000 requests

then:

$$
10,000,000 \times 0.01 = 100,000
$$

requests experience the bad latency.

That is why serious systems care about:

p50
p90
p95
p99
p99.9

The tail tells you what happens when the system is under pressure.

And database performance often lives in the tail.

17. Connection Pools Are Queues

Your application probably has a database connection pool.

Suppose you configure:

20 connections

and receive:

100 concurrent database operations

Only 20 can actively use connections at once.

The remaining 80 wait.

You now have a queue.

Increasing the connection pool can help.

But there is a trap.

If your database can efficiently process only 50 concurrent operations, increasing the pool to:

does not magically create 500 units of database capacity.

It may simply create:

more contention
more memory usage
more context switching
more lock competition
more queueing

This is another lesson from queueing theory:

More concurrency is not automatically more throughput.

At some point, you hit the service capacity of the database.

18. Amdahl's Law: Why One Slow Part Dominates

Suppose a request spends:

80% database
20% application logic

You optimize the application logic and make it 10x faster.

What happens?

The application portion goes from:

20 units → 2 units

The database remains:

80 units

Total:

82 units

Original:

100 units

Speedup:

$$
\frac{100}{82} \approx 1.22
$$

You made one component 10x faster.

The entire system improved only about:

22%

This is Amdahl's Law.

The basic equation is:

$$
Speedup = \frac{1}{(1-p)+\frac{p}{s}}
$$

Where:

(p) = fraction of work improved
(s) = speedup of that component

This is why database optimization matters so much when the database dominates request latency.

And it also explains why optimizing irrelevant code is often a waste of time.

19. Database Performance Is Usually About Reducing Work

There is a recurring theme across everything we've discussed.

Indexes reduce search work.

Query predicates reduce rows.

Pagination reduces returned work.

Caching reduces database work.

Batching reduces round trips.

Prepared statements reduce repeated parsing work.

Materialized views reduce repeated computation.

Denormalization can reduce join work.

Partitioning reduces the amount of data considered.

The common principle is:

$$
Performance \approx \frac{Useful\ Work}{Total\ Work}
$$

Or more practically:

The fastest query is often the query that does not have to do something.

This is why performance engineers ask:

Why are we reading this row?
Why are we joining this table?
Why are we sorting this?
Why are we fetching this column?
Why are we making this request?
Why are we doing this query 100 times?

Optimization begins with subtraction.

20. The Mathematics of Data Growth

Suppose your database starts with:

100,000 rows

and grows by:

1,000,000 rows/month

After one year:

$$
100,000 + 12(1,000,000)
$$

$$
= 12,100,000
$$

Your query might work perfectly at 100,000 rows.

But if its cost is:

$$
O(n)
$$

then the amount of work scales directly with the data.

If the cost is:

$$
O(n\log n)
$$

growth becomes even more interesting.

If the cost is:

$$
O(n^2)
$$

you are eventually going to have a very bad day.

This is why complexity analysis matters even in SQL.

A query that looks harmless against a development database may be algorithmically disastrous against production data.

Your database size today is not the only thing that matters.

You need to ask:

What happens when the data is 100x larger?

21. Partitioning: Divide the Problem

Suppose a database contains:

10 billion events

but your application usually queries only the current month.

Why search through 10 billion rows?

Partitioning can divide the dataset:

Now a query such as:

WHERE created_at >= '2026-08-01'

can potentially eliminate partitions that cannot contain relevant data.

This is called partition pruning.

Mathematically, instead of searching:

$$
N
$$

rows, you may search approximately:

$$
N_k
$$

where:

$$
N_k \ll N
$$

This is the same fundamental strategy used everywhere in computer science:

Reduce the search space.

Binary search does it.

Indexes do it.

Hashing does it.

Partitioning does it.

Caching does it.

Distributed databases do it.

Good algorithms spend less time looking in places where the answer cannot possibly exist.

22. Normalization Has a Mathematical Cost

Database normalization is fantastic.

It reduces duplication.

It improves consistency.

But normalization can increase the number of joins required to reconstruct information.

Suppose a customer profile is distributed across:

customers
addresses
preferences
subscriptions
orders

A dashboard may need several joins.

If the application repeatedly needs the same combined representation, you might consider:

denormalization
materialized views
caching
precomputed aggregates

This is not an argument against normalization.

It is an argument for understanding trade-offs.

You are exchanging:

write complexity / storage duplication

for:

read complexity / join cost

Database architecture is full of these equations.

There is rarely a universally optimal design.

There is usually an optimization problem.

23. The Hidden Cost of Network Round Trips

Suppose your application and database are separated by:

1 ms

of network latency.

One query:

1 ms

Not terrible.

But now your application performs:

100 queries

sequentially.

Ignoring database execution time:

$$
100 \times 1ms = 100ms
$$

If network latency is:

5 ms

then:

$$
100 \times 5ms = 500ms
$$

The database might be incredibly fast.

Your architecture can still be slow.

This is why batching is powerful.

Instead of:

100 round trips

you might perform:

1 round trip

and process the batch server-side.

This is another case where multiplication destroys performance.

24. Why "SELECT *" Is More Than a Style Problem

Consider:

SELECT *
FROM users
WHERE id = 123;

It works.

But suppose the table contains:

id
name
email
profile
preferences
avatar
biography
settings
large_json_blob

You only need:

id
name
email

Yet the database may have to read and transfer more information than necessary.

The cost of a query is influenced by:

$$
Rows \times Columns \times RowWidth
$$

If you reduce row width, you reduce:

I/O
memory bandwidth
network transfer
serialization
deserialization

This is why data size matters.

The database is not moving abstract objects.

It is moving bytes.

25. Write Performance Has Its Own Mathematics

Everyone loves talking about read performance.

Then production gets busy.

Writes become the problem.

Suppose one insert modifies:

1 table

with:

```text 5 indexes




The database does not merely insert the row.

It must maintain those indexes.

Conceptually:



```text
Insert row
   ↓
Update index 1
   ↓
Update index 2
   ↓
Update index 3
   ↓
Update index 4
   ↓
Update index 5

More indexes can make reads faster while making writes more expensive.

So indexing is not:

free speed

It is a trade-off.

You are effectively paying:

write cost + storage + maintenance

to obtain:

read performance

This is why every index should have a reason to exist.

26. Transactions and Concurrency

Now we reach one of the hardest parts.

Concurrency.

Imagine two transactions both attempt:

balance = balance - 100

at the same time.

Without proper isolation, you can get race conditions.

Databases therefore implement mechanisms involving:

locks
MVCC
snapshots
isolation levels
conflict detection

These mechanisms have performance costs.

Higher isolation can reduce certain anomalies but may increase contention.

More locking can mean more waiting.

More waiting means greater latency.

And again:

$$
Concurrency \rightarrow Queueing \rightarrow Latency
$$

Everything connects.

Database performance cannot be separated from correctness.

A database that is extremely fast but produces incorrect balances is not performant.

It is broken.

27. The Database Optimizer Is a Statistician

Modern query optimizers do not blindly execute SQL in the written order.

They estimate.

They ask:

How many rows will this predicate return?

How selective is this condition?

How expensive is this join?

Which index is useful?

Should I hash?

Should I sort?

Should I scan?

These decisions depend heavily on statistics.

Imagine:

10 million rows

but only:

10 rows

match a predicate.

An index is likely attractive.

Now imagine:

9 million rows

match.

A sequential scan may be cheaper.

The optimizer is essentially trying to estimate:

$$
Expected\ Cost
$$

for different execution plans.

This is why stale statistics can sometimes lead to terrible query plans.

The optimizer is solving a problem using information.

Bad information produces bad decisions.

28. Performance Is an Optimization Function

You can think of database design as optimizing something like:

$$
Cost =
C_{CPU}
+
C_{IO}
+
C_{Memory}
+
C_{Network}
+
C_{Contention}
$$

The exact formulas differ between database engines.

But the conceptual model is useful.

A query can be CPU-heavy.

Or I/O-heavy.

Or network-heavy.

Or memory-heavy.

Or blocked by locks.

That means "optimize the query" is not a complete engineering instruction.

You need to ask:

Which resource is actually limiting us?

If the query is CPU-bound, adding RAM may do almost nothing.

If it is I/O-bound, optimizing a JavaScript loop is irrelevant.

If it is network-bound, rewriting the SQL may not help.

Performance engineering begins with measurement.

29. Why EXPLAIN Is Basically a Mathematical Window

When you run:

EXPLAIN ANALYZE

you are asking the database:

"Show me the computational strategy you chose."

You might see:

Seq Scan
Index Scan
Nested Loop
Hash Join
Sort
Aggregate

These are algorithms.

And their costs depend on data size.

For example:

Seq Scan
Rows Removed by Filter: 9,900,000
Rows Returned: 100,000

That tells a story.

The database examined a huge amount of data to produce a smaller result.

You can then ask:

Can I make the database avoid touching those 9.9 million rows?

That is the mathematical question hiding behind the query plan.

30. The Most Dangerous Complexity Is the Complexity You Cannot See

A database query can look like:

SELECT *
FROM orders
WHERE customer_id = ?

One line.

But internally it may involve:

parse
↓
plan
↓
index lookup
↓
page reads
↓
buffer cache
↓
visibility checks
↓
row reconstruction
↓
serialization
↓
network transfer

The SQL is only the tip of the iceberg.

This is why database engineering is so fascinating.

A declarative language allows you to describe what you want.

The database decides how to obtain it.

That "how" is where algorithms and mathematics live.

31. Database Performance Is Applied Computer Science

Once you understand the mathematics, database optimization stops feeling like folklore.

You begin seeing familiar concepts everywhere.

Indexes

$$
O(\log n)
$$

Hash joins

Approximately:

$$
O(n+m)
$$

Sorting

$$
O(n\log n)
$$

Naive nested loops

$$
O(nm)
$$

Pagination

Offset:

$$
O(offset+k)
$$

Cursor-based:

$$
O(\log n+k)
$$

Caching

Expected latency:

$$
E[T] =
P(hit)T_{hit}
+
P(miss)T_{miss}
$$

Concurrency

Little's Law:

$$
L=\lambda W
$$

Optimization

Amdahl's Law:

$$
S=\frac{1}{(1-p)+p/s}
$$

These aren't just academic equations.

They show up in production.

They show up when your API suddenly gets slow.

They show up when your dashboard times out.

They show up when your database CPU reaches 100%.

They show up when adding an index makes writes slower.

They show up when pagination collapses at page 10,000.

They show up when a cache miss causes a latency spike.

32. The Ultimate Database Question

Whenever you encounter a slow query, resist the temptation to immediately change something.

Instead ask:

What is the database actually doing?

Then ask:

How much data is it touching?

Then:

How many rows survive each filtering step?

Then:

What algorithm is being used?

Then:

Where is the latency coming from?

Then:

What happens when the dataset grows 10x?

Then:

What happens under 100x concurrency?

These questions lead you toward the real problem.

Not:

"Should I add an index?"

But:

"What computational problem am I asking the database to solve?"

That is a much better question.

33. The Database Is a Machine for Avoiding Work

This might be the deepest idea in the entire article.

Good database systems are fundamentally machines for avoiding unnecessary work.

An index says:

Don't scan everything.

A cache says:

Don't calculate this again.

A materialized view says:

Don't perform this expensive join every time.

Partitioning says:

Don't search irrelevant data.

A query predicate says:

Don't return unnecessary rows.

Projection says:

Don't move unnecessary columns.

Batching says:

Don't pay network latency repeatedly.

A good schema says:

Don't create ambiguity that forces expensive operations.

The entire field of database optimization can almost be summarized as:

Do less work, move fewer bytes, wait less, and make the remaining work predictable.

That is the mathematics.

Final Thoughts

The next time somebody tells you:

"The database is slow."

Do not immediately believe them.

The database may be slow.

But something else may also be happening.

Maybe the application is executing 101 queries instead of 2.

Maybe an index has terrible selectivity.

Maybe pagination is scanning a million rows.

Maybe a query is transferring megabytes to retrieve three fields.

Maybe a connection pool is creating massive contention.

Maybe the optimizer has bad statistics.

Maybe the system is suffering from tail latency.

Maybe the database is waiting on storage.

Maybe the database is waiting on locks.

Maybe the database is actually fine.

The interesting part is that almost all of these problems can be expressed mathematically.

That is what makes database engineering so beautiful.

Behind:

SELECT

there is complexity.

Behind:

JOIN

there is an algorithm.

Behind:

WHERE

there is selectivity.

Behind:

ORDER BY

there is sorting or ordering.

Behind:

LIMIT

there is a question about how much data must actually be produced.

Behind:

INDEX

there is logarithmic search.

Behind:

CACHE

there is probability.

Behind:

CONNECTION POOL

there is queueing theory.

Behind:

TRANSACTION

there is concurrency theory.

And behind all of it is one simple engineering principle:

Performance is what happens when computation meets scale.

At small scale, almost every design looks intelligent.

At large scale, mathematics starts asking questions.

How many rows?

How many requests?

How many bytes?

How many joins?

How many round trips?

How many concurrent users?

How many milliseconds?

How many times are we doing the same work?

And eventually the database stops being a mysterious black box.

You start seeing it for what it really is.

A collection of algorithms operating over enormous mathematical structures, constrained by physical hardware, probabilities, queues, memory hierarchies, and the laws of computation.

The SQL is merely the language you use to ask the question.

The database's job is to find the cheapest way to answer it.

And your job as an engineer is to understand why that answer costs what it costs.

Because once you understand the mathematics behind database performance, you stop optimizing by superstition.

You stop adding indexes because someone on Stack Overflow said so.

You stop throwing RAM at every problem.

You stop blaming the database for application-level inefficiencies.

You start measuring.

You start modeling.

You start thinking in:

$$
O(n)
$$

$$
O(\log n)
$$

$$
O(n\log n)
$$

$$
O(nm)
$$

You start thinking about:

$$
Latency
$$

$$
Throughput
$$

$$
Selectivity
$$

$$
Probability
$$

$$
Contention
$$

$$
Bandwidth
$$

$$
Queueing
$$

And suddenly database performance becomes much less mysterious.

Because the database was never magic.

It was mathematics the entire time.

Encoding State Machines in the Type System

Derek Mwale — Sat, 29 Aug 2026 09:14:23 +0000

What if invalid states were not bugs you had to test for—but programs you could not compile?

Most software is secretly a collection of state machines.

A user is logged out, then logged in.

An order is created, then paid for, then shipped, then delivered.

A network connection is disconnected, connecting, connected, or closed.

A file is unopened, opened, written to, and eventually closed.

A database transaction is started, committed, or rolled back.

A rocket is idle, armed, launched, and hopefully not exploded.

The funny thing is that we rarely write software as if these are state machines.

Instead, we create an object.

We give it a status field.

Then we spend the next five years writing if statements to stop people from doing things they should never have been able to do in the first place.

if order.status == "paid" {
    ship_order(order);
}

Then someone ships the order twice.

So we add another condition.

if order.status == "paid" && !order.shipped {
    ship_order(order);
}

Then refunds appear.

Then cancellations.

Then partial shipments.

Then retries.

Then asynchronous workers.

Then distributed systems.

Then suddenly your innocent little status: String has become a crime scene.

This is where one of the most powerful ideas in modern programming enters the conversation:

What if we encoded the state machine directly into the type system?

Instead of asking at runtime:

“Is this object currently allowed to do this?”

We ask at compile time:

“Can this operation even be expressed on this type?”

That shift sounds small.

It is not.

It changes how you design APIs.

It changes how you think about correctness.

It changes where bugs are allowed to exist.

And Rust, with its ownership system, enums, generics, traits, and zero-cost abstractions, happens to be an absolutely fascinating language for exploring this idea.

Welcome to the world where types do not merely describe data.

They describe time.

They describe history.

They describe what has already happened.

And sometimes, they describe what is allowed to happen next.

The Hidden State Machine Inside Your Code

Let's begin with something extremely ordinary.

Imagine a payment system.

A payment can be:

Created
Authorized
Captured
Refunded
Cancelled

We might represent it like this:

struct Payment {
    id: String,
    status: String,
}

Immediately, the program has a problem.

The type Payment represents all possible states.

A payment that has just been created and a payment that has already been refunded are both:

Payment

The compiler cannot tell them apart.

So this becomes possible:

refund(payment);

Even if the payment was never captured.

Or:

capture(payment);
capture(payment);

Even if capturing twice makes absolutely no sense.

The type system sees only:

Payment

Humans see:

Created → Authorized → Captured → Refunded

That difference is where bugs live.

Traditional programming solves this by storing state as data:

enum PaymentStatus {
    Created,
    Authorized,
    Captured,
    Refunded,
    Cancelled,
}

struct Payment {
    id: String,
    status: PaymentStatus,
}

This is already much better.

But the fundamental issue remains.

Every function still accepts:

Payment

So every operation must defend itself.

fn capture(payment: &mut Payment) -> Result<(), Error> {
    match payment.status {
        PaymentStatus::Authorized => {
            payment.status = PaymentStatus::Captured;
            Ok(())
        }
        _ => Err(Error::InvalidState),
    }
}

This is runtime validation.

The program can still attempt something illegal.

It just discovers the illegality later.

The compiler allows the program.

The program runs.

The function checks the state.

The function returns an error.

This works.

But there is another way.

Instead of representing state as a value inside an object, we can represent state as part of the object's type.

From Runtime States to Compile-Time States

Imagine these types:

struct CreatedPayment {
    id: String,
}

struct AuthorizedPayment {
    id: String,
}

struct CapturedPayment {
    id: String,
}

struct RefundedPayment {
    id: String,
}

Now our transitions become functions:

fn authorize(payment: CreatedPayment) -> AuthorizedPayment {
    AuthorizedPayment {
        id: payment.id,
    }
}

Then:

fn capture(payment: AuthorizedPayment) -> CapturedPayment {
    CapturedPayment {
        id: payment.id,
    }
}

And:

fn refund(payment: CapturedPayment) -> RefundedPayment {
    RefundedPayment {
        id: payment.id,
    }
}

Look carefully at what just happened.

The API itself now describes the state machine.

CreatedPayment
        |
        | authorize()
        v
AuthorizedPayment
        |
        | capture()
        v
CapturedPayment
        |
        | refund()
        v
RefundedPayment

And now this code does not compile:

let payment = CreatedPayment {
    id: "123".into(),
};

let refunded = refund(payment);

Why?

Because refund() requires:

CapturedPayment

But we have:

CreatedPayment

This is not a runtime error.

There is no Result.

There is no exception.

There is no if.

There is no test required to discover that this transition is invalid.

The program simply cannot be compiled.

That is an incredibly powerful property.

We have moved a category of bugs from:

“Something that might happen in production.”

to:

“Something that cannot be represented as a valid program.”

The Compiler Becomes Your State Machine Enforcer

This is the mental shift that makes typestate programming so interesting.

Normally, a compiler answers questions like:

Does this variable exist?
Are the types compatible?
Are we borrowing safely?
Are we accessing valid memory?

With state encoded into types, the compiler starts answering another question:

Is this operation legal at this point in the lifecycle?

Imagine a TCP connection.

A normal object-oriented representation might look like this:

struct Connection {
    connected: bool,
}

Then:

fn send(connection: &Connection, data: &[u8]) {
    if !connection.connected {
        panic!("Not connected");
    }

    // send data
}

The state is checked at runtime.

But what if we instead had:

struct Disconnected;

struct Connected {
    socket: Socket,
}

Then:

impl Disconnected {
    fn connect(self) -> Connected {
        Connected {
            socket: Socket::new(),
        }
    }
}

And:

impl Connected {
    fn send(&self, data: &[u8]) {
        // send data
    }

    fn disconnect(self) -> Disconnected {
        Disconnected
    }
}

Now the lifecycle becomes visible in the API:

let connection = Disconnected;

let connection = connection.connect();

connection.send(b"Hello, world!");

let connection = connection.disconnect();

But this is impossible:

let connection = Disconnected;

connection.send(b"Hello");

There is no send() method.

Not because we wrote an if statement.

Not because the network rejected it.

Not because a test caught it.

Because the type does not expose the operation.

This is one of the cleanest forms of API design possible.

If you cannot do something, the method does not exist.

Types Can Represent Time

This is where things become slightly philosophical.

Most programmers think types represent structure.

For example:

String
u32
User
Order
Vec<T>

They describe what something is.

But types can also describe where something is in a process.

Consider:

DraftPost

versus:

PublishedPost

Both might contain:

title
content
author

But they are not semantically identical.

One can be edited.

The other might be immutable.

One can be published.

The other cannot.

One can be deleted without consequences.

The other might already have readers.

The data may look similar.

The meaning is completely different.

That means state is not merely data.

State is context.

And context changes what operations are valid.

This is why encoding state into types is so powerful.

The type becomes a description of both:

What this thing is

and:

Where this thing is in its lifecycle

Your program starts gaining a memory of its own history.

The Typestate Pattern

This design approach is commonly known as the Typestate Pattern.

The basic idea is simple:

Represent the state of an object using its type.

Let's build something more realistic.

Imagine an HTTP request builder.

Normally, we might do this:

struct RequestBuilder {
    url: Option<String>,
    method: Option<String>,
}

Then:

impl RequestBuilder {
    fn send(self) -> Result<Response, Error> {
        let url = self.url.ok_or(Error::MissingUrl)?;
        let method = self.method.ok_or(Error::MissingMethod)?;

        // send request
    }
}

Again, we are validating required state at runtime.

But suppose we encode the builder's completeness into its type.

struct Missing;

struct Present;

Then:

struct RequestBuilder<UrlState, MethodState> {
    url: Option<String>,
    method: Option<String>,
    _url: std::marker::PhantomData<UrlState>,
    _method: std::marker::PhantomData<MethodState>,
}

Now creating a builder gives us:

RequestBuilder<Missing, Missing>

Setting the URL changes the type:

impl<M> RequestBuilder<Missing, M> {
    fn url(self, url: String) -> RequestBuilder<Present, M> {
        RequestBuilder {
            url: Some(url),
            method: self.method,
            _url: std::marker::PhantomData,
            _method: std::marker::PhantomData,
        }
    }
}

Setting the method does the same:

impl<U> RequestBuilder<U, Missing> {
    fn method(self, method: String) -> RequestBuilder<U, Present> {
        RequestBuilder {
            url: self.url,
            method: Some(method),
            _url: std::marker::PhantomData,
            _method: std::marker::PhantomData,
        }
    }
}

And now we only implement send() for a fully configured builder.

impl RequestBuilder<Present, Present> {
    fn send(self) -> Response {
        Response
    }
}

Now the compiler understands the lifecycle.

This works:

let request = RequestBuilder::new()
    .url("https://example.com".into())
    .method("GET".into())
    .send();

But this does not:

let request = RequestBuilder::new()
    .url("https://example.com".into())
    .send();

The method does not exist for:

RequestBuilder<Present, Missing>

That is beautiful.

We did not write:

if method.is_none()

We did not return:

Result<Response, Error>

We changed the problem.

Instead of checking whether the object is valid, we made it impossible to call send() until the object becomes the correct type.

`PhantomData`: Encoding Information That Does Not Exist at Runtime

The interesting thing about the previous example is this:

UrlState

and:

MethodState

do not necessarily exist as actual runtime values.

They exist for the compiler.

This is where Rust's:

PhantomData<T>

becomes extremely useful.

PhantomData basically tells the compiler:

“This type logically depends on T, even though no actual value of T is stored here.”

That means:

RequestBuilder<Present, Missing>

and:

RequestBuilder<Present, Present>

can have the same runtime memory layout.

But the compiler treats them as different types.

This is one of the magical things about type-level state.

You can gain stronger correctness guarantees without necessarily paying runtime costs.

The state machine can disappear during compilation.

The runtime does not need to carry a giant state machine around.

The compiler already used it to prove certain things.

Illegal States Should Be Unrepresentable

There is a famous principle in type-driven programming:

Make illegal states unrepresentable.

This sounds like a slogan.

But it is actually a design philosophy.

Consider a user authentication system.

A poorly designed API might look like this:

struct User {
    id: String,
    authenticated: bool,
    token: Option<String>,
}

Now we can have weird combinations:

authenticated = true
token = None

Or:

authenticated = false
token = Some(...)

Are these valid?

Maybe.

Maybe not.

The problem is that the structure allows combinations that might make no semantic sense.

Instead, we can model the states directly.

struct AnonymousUser {
    id: String,
}

struct AuthenticatedUser {
    id: String,
    token: String,
}

Now the invalid combinations disappear.

A function requiring authentication can accept:

AuthenticatedUser

fn access_dashboard(user: &AuthenticatedUser) {
    println!("Welcome {}", user.id);
}

You cannot accidentally pass:

AnonymousUser

The authorization requirement is now visible in the function signature.

That matters enormously.

Because function signatures are documentation.

But unlike comments, they cannot lie as easily.

The State Machine Is the API

When you encode state transitions into types, something fascinating happens.

Your API starts becoming a diagram.

Consider a deployment pipeline.

Code
 ↓
Built
 ↓
Tested
 ↓
Approved
 ↓
Deployed

We can represent this as types:

struct Code;

struct Built;

struct Tested;

struct Approved;

struct Deployed;

Then:

fn build(_: Code) -> Built {
    Built
}

fn test(_: Built) -> Tested {
    Tested
}

fn approve(_: Tested) -> Approved {
    Approved
}

fn deploy(_: Approved) -> Deployed {
    Deployed
}

Now look at the program:

let code = Code;

let built = build(code);
let tested = test(built);
let approved = approve(tested);
let deployed = deploy(approved);

The source code itself tells the story.

And more importantly, the compiler guarantees the order.

This does not compile:

let code = Code;

let deployed = deploy(code);

There is no runtime rule required.

The function signature itself defines the transition graph.

You have essentially encoded this:

Code → Built → Tested → Approved → Deployed

into the type system.

Ownership Makes State Transitions Feel Natural

Rust has a particularly interesting advantage here.

Ownership naturally models transitions.

Look at this:

fn authorize(payment: CreatedPayment) -> AuthorizedPayment

The old state is consumed.

The new state is produced.

That is not accidental.

It perfectly matches the semantics of a state transition.

Before:

CreatedPayment

After:

AuthorizedPayment

You do not have both.

The previous state has been moved.

This prevents a surprisingly common class of mistakes.

Imagine this:

let payment = CreatedPayment {
    id: "123".into(),
};

let authorized = authorize(payment);

// payment is gone

You cannot continue using the old CreatedPayment.

Rust's ownership system enforces the timeline.

This is deeper than ordinary type checking.

The type system tells you:

What state you are in.

Ownership tells you:

You cannot pretend you are still in the previous state.

That combination is ridiculously powerful.

Linear Logic Hiding Inside Your Rust Code

Without getting too academic, there is a connection here to an idea from logic and mathematics called linear logic.

Traditional variables can often be copied or reused freely.

But some resources should not behave like that.

Examples:

A database transaction
A file handle
A network socket
A lock
A cryptographic secret
A payment authorization
A unique ownership token

These things have lifecycles.

You should not casually duplicate them.

A database transaction is a perfect example.

You might have:

struct ActiveTransaction;

struct CommittedTransaction;

struct RolledBackTransaction;

Then:

fn commit(tx: ActiveTransaction) -> CommittedTransaction

or:

fn rollback(tx: ActiveTransaction) -> RolledBackTransaction

Once committed:

let committed = commit(tx);

You cannot also roll it back.

Because tx was consumed.

That means this is impossible:

let committed = commit(tx);
let rolled_back = rollback(tx);

The compiler says:

You already used that resource.

Which is exactly what we want.

A transaction should not simultaneously be committed and rolled back.

Reality itself does not support that state.

Why should your type system?

Enums vs Typestate

At this point, you might ask:

Why not just use enums?

Excellent question.

Enums are fantastic.

Consider:

enum Order {
    Created {
        id: String,
    },

    Paid {
        id: String,
    },

    Shipped {
        id: String,
    },

    Delivered {
        id: String,
    },
}

This is a very strong representation.

It prevents impossible data combinations.

But enums and typestate solve slightly different problems.

With enums, one variable can represent multiple possible states:

Order

You then pattern match:

match order {
    Order::Created { .. } => {}
    Order::Paid { .. } => {}
    Order::Shipped { .. } => {}
    Order::Delivered { .. } => {}
}

Typestate instead often creates separate types:

CreatedOrder
PaidOrder
ShippedOrder
DeliveredOrder

Then functions can restrict themselves to specific states.

fn ship(order: PaidOrder) -> ShippedOrder

The difference is subtle but important.

Enums say:

“This value may currently be one of these states.”

Typestate says:

“This function only accepts this exact state.”

In practice, the two approaches can complement each other.

Enums are excellent when you need to inspect dynamic state.

Typestate is excellent when you want to constrain an API.

The question is not:

Which one is better?

The question is:

Where do you want correctness to live?

A File API That Cannot Forget to Close

Let's imagine designing a file abstraction.

A naive API:

struct File {
    handle: Option<Handle>,
}

Then:

fn read(file: &File) -> Result<Vec<u8>, Error> {
    let handle = file.handle.as_ref()
        .ok_or(Error::FileClosed)?;

    // read
}

Again, runtime state checking.

Now consider typestate:

struct ClosedFile {
    path: String,
}

struct OpenFile {
    handle: Handle,
}

Then:

impl ClosedFile {
    fn open(self) -> OpenFile {
        OpenFile {
            handle: open_handle(self.path),
        }
    }
}

And:

impl OpenFile {
    fn read(&self) -> Vec<u8> {
        // read
        vec![]
    }

    fn close(self) -> ClosedFile {
        ClosedFile {
            path: "file.txt".into(),
        }
    }
}

Now:

let file = ClosedFile {
    path: "data.txt".into(),
};

let file = file.open();

let data = file.read();

let file = file.close();

You cannot read after closing because:

ClosedFile

does not expose:

read()

The lifecycle is enforced by the API.

This is the type system acting like a traffic controller.

You may proceed.

You may not proceed.

And it makes that decision before your program runs.

State Machines Become Composable

The really interesting part begins when we combine typestate with generics.

Imagine a generic document:

struct Draft;

struct Review;

struct Published;

Then:

struct Document<State> {
    title: String,
    content: String,
    _state: std::marker::PhantomData<State>,
}

Now we can implement operations selectively.

impl Document<Draft> {
    fn submit_for_review(self) -> Document<Review> {
        Document {
            title: self.title,
            content: self.content,
            _state: std::marker::PhantomData,
        }
    }
}

Then:

impl Document<Review> {
    fn publish(self) -> Document<Published> {
        Document {
            title: self.title,
            content: self.content,
            _state: std::marker::PhantomData,
        }
    }
}

And:

impl Document<Published> {
    fn public_url(&self) -> String {
        format!("https://example.com/{}", self.title)
    }
}

The generic container:

Document<State>

represents the concept.

The generic parameter represents the lifecycle position.

This scales beautifully.

You are not creating entirely unrelated structures.

You are saying:

“A document exists in different compile-time states.”

That is a very elegant abstraction.

But Can You Overdo This?

Absolutely.

And this is important.

Typestate programming can become a type-level fever dream.

You start with:

Document<Draft>

Then:

Document<Draft, Unsigned, Unapproved, Unencrypted>

Then:

Document<
    Draft,
    Unsigned,
    Unapproved,
    Unencrypted,
    Local,
    Uncompressed,
    Unindexed
>

Congratulations.

You have successfully turned your application into a doctoral thesis.

The compiler is powerful.

But humans still need to read the code.

The goal is not:

Encode every possible fact into the type system.

The goal is:

Encode important invariants where doing so makes invalid behavior impossible or significantly harder.

Typestate is most valuable when:

State transitions are strict.
Invalid transitions are dangerous.
Runtime checks are repetitive.
The lifecycle is central to the API.
Resources have ownership semantics.
Bugs are expensive.

It might be unnecessary for:

Simple CRUD entities.
Highly dynamic workflows.
States determined entirely by external systems.
Rapidly changing business rules.

There is always a trade-off.

Strong types are not free.

They cost conceptual complexity.

The question is whether that complexity is worth buying.

Distributed Systems Are Full of State Machines

This idea becomes even more interesting when we move beyond local programs.

Distributed systems are basically giant state machines wearing trench coats.

A message can be:

Created
Queued
Delivered
Acknowledged
Failed
Retried
Dead-Lettered

A service can be:

Starting
Healthy
Degraded
Unavailable
Recovering

A consensus node can be:

Follower
Candidate
Leader

A transaction can be:

Prepared
Committed
Aborted

And distributed systems are notoriously difficult because invalid transitions happen across:

machines
networks
processes
time
failures

You cannot encode the entire distributed world into Rust's type system.

The network does not care about your generics.

A remote server can always behave unexpectedly.

But you can encode your local protocol guarantees.

For example, a client library might ensure that:

Handshake must happen before authentication.
Authentication must happen before requests.
Shutdown prevents future requests.

The network remains uncertain.

But your API becomes safer.

This is a crucial distinction.

Types cannot eliminate uncertainty.

But they can eliminate nonsense.

And eliminating nonsense is already an incredible engineering achievement.

Session Types: Typestate's Bigger Cousin

If typestate describes the lifecycle of an object, there is an even more fascinating concept called session types.

Session types attempt to describe communication protocols.

Imagine a protocol:

Client → Server: Hello
Server → Client: Challenge
Client → Server: Response
Server → Client: Success

A session type can describe what communication is expected next.

Conceptually:

Send Hello
↓
Receive Challenge
↓
Send Response
↓
Receive Success

The type system can then help ensure you do not send messages in the wrong order.

This is where programming starts feeling almost futuristic.

The compiler is no longer just checking data structures.

It is checking conversations.

It is checking protocols.

It is checking sequences of events.

That idea should make every backend engineer slightly excited.

Because how many production bugs are really just:

“Two systems disagreed about what was supposed to happen next.”

The Type System as a Proof Engine

There is another way to think about all of this.

A program is not only a sequence of instructions.

A program can also be a proof.

When we write:

fn capture(payment: AuthorizedPayment) -> CapturedPayment

we are making a claim:

A captured payment can only be produced from an authorized payment.

The compiler checks whether our code obeys the rules required to make that claim meaningful.

If a function requires:

AuthenticatedUser

then calling that function is evidence that authentication has already happened somewhere in the program's control flow.

The type becomes a proof of a previous event.

That is mind-bending.

Consider:

fn deploy(app: TestedApplication)

The existence of:

TestedApplication

is evidence that testing happened.

Or at least that the only legitimate way to construct that type requires testing.

This is where API design becomes incredibly important.

If developers can casually construct:

TestedApplication

then the guarantee is meaningless.

So constructors matter.

Visibility matters.

Modules matter.

The strength of the state machine depends on controlling how states are created.

Smart Constructors and Protected Transitions

Imagine this:

pub struct AuthorizedPayment {
    id: String,
}

Anyone could potentially create one.

That defeats the purpose.

Instead, we can make internal fields private:

pub struct AuthorizedPayment {
    id: String,
}

Then only expose transitions.

impl CreatedPayment {
    pub fn authorize(self) -> AuthorizedPayment {
        AuthorizedPayment {
            id: self.id,
        }
    }
}

The outside world cannot simply say:

AuthorizedPayment {
    id: "123".into(),
};

if the fields are private.

Now the state becomes meaningful.

You have created a controlled state transition system.

The only paths through the state machine are the paths exposed by the API.

That is exactly what we want.

Error Handling Still Exists

A common misunderstanding is that encoding state into types removes errors.

It does not.

It removes certain categories of errors.

Suppose:

fn connect(self) -> Connected

In reality, a network connection might fail.

So the real signature becomes:

fn connect(self) -> Result<Connected, ConnectionError>

The type system guarantees:

You cannot send data before connecting.

But it cannot guarantee:

The network will always allow you to connect.

Those are different things.

Types are excellent at modeling logical impossibilities.

They cannot eliminate physical uncertainty.

For example:

Sending before connecting

is logically invalid.

But:

Connection timed out

is a real-world failure.

The first can be eliminated through types.

The second requires runtime error handling.

Great engineering comes from knowing which category you are dealing with.

The Most Interesting API Is Often the One That Lets You Do Less

We often judge APIs by how much they can do.

But sometimes the best API is defined by what it refuses to let you do.

A good API might say:

You cannot:

Commit a transaction twice.
Read a closed file.
Send through a disconnected socket.
Publish an incomplete article.
Deploy unapproved code.
Spend an already-consumed token.
Access a dashboard without authentication.

That is not a limitation.

That is intelligence.

The API understands its own rules.

And when the rules are embedded into types, developers do not have to remember everything.

The compiler remembers.

This is one of the most underrated benefits of strong typing.

Humans are unreliable state machines.

We forget.

We misunderstand.

We copy-paste code.

We call functions in the wrong order.

We assume.

We are tired.

The compiler is not tired.

The compiler does not care that it is 2 AM.

The compiler does not say:

“I think this transition is probably fine.”

It either accepts the program or rejects it.

And honestly, sometimes we need that kind of friend.

State Machines Are Everywhere

Once you start seeing state machines, you cannot stop.

Your login flow is a state machine.

Your shopping cart is a state machine.

Your CI/CD pipeline is a state machine.

Your database connection is a state machine.

Your Kubernetes pod lifecycle is a state machine.

Your compiler is a state machine.

Your music player is a state machine.

Your coffee machine is probably a state machine.

Even your own software development process looks suspiciously like:

Idea
↓
Prototype
↓
Buggy
↓
Very Buggy
↓
Rewrite
↓
Production
↓
Emergency

We live inside transitions.

Programming is largely the art of deciding:

What can happen next?

And that is exactly what a state machine describes.

The type system gives us a way to move that question earlier.

Instead of discovering the answer while the program is running, we can encode parts of the answer into the program's structure.

The Future of Correctness Is Moving Left

Software engineering has spent decades moving problems earlier.

We moved from:

Production bug

to:

Runtime error

Then to:

Test failure

Then to:

Static analysis warning

Then to:

Compiler error

Every step leftward is cheaper.

A production bug might cost money.

A failed test costs developer time.

A compiler error costs a few seconds.

Encoding state machines in types is part of that movement.

We are taking business and lifecycle rules that would normally be enforced during execution and asking:

Can the compiler help us enforce these before execution?

Sometimes the answer is no.

But sometimes the answer is beautifully yes.

The Real Lesson

The biggest lesson here is not:

“Use PhantomData everywhere.”

Please do not.

The real lesson is much bigger.

When designing software, ask:

What states can this thing exist in?

Then ask:

Which transitions are valid?

Then:

Which transitions are impossible?

And finally:

Can the type system represent those rules?

If the answer is yes, you may be able to design an API where entire categories of bugs disappear.

Not because you tested harder.

Not because you wrote better comments.

Not because your developers became perfect.

But because the invalid program stopped being expressible.

That is one of the deepest ideas in programming.

The type system is not merely there to complain when you add a string to an integer.

It can model constraints.

It can model protocols.

It can model lifecycles.

It can model history.

And in the right hands, it can model the flow of time itself.

Final Thoughts

State machines are one of those concepts that seem simple when you first encounter them.

A few circles.

A few arrows.

A few transitions.

Then you start building real systems and realize that almost everything important is governed by state.

The question is not whether your application has state machines.

It does.

The question is where those state machines live.

Do they live in:

if statements?

Do they live in:

status == "something"

Do they live in documentation nobody reads?

Do they live in the heads of senior engineers?

Or do they live somewhere much more reliable?

The type system.

Encoding state machines into types is ultimately about making software more honest.

A disconnected connection should not look like a connected connection.

An unauthenticated user should not look like an authenticated user.

An unapproved deployment should not look like a deployable application.

A closed file should not expose read().

A payment that has never been captured should not be refundable.

The closer your types match reality, the less your program has to constantly defend itself from impossible situations.

And that might be one of the most beautiful goals in software engineering:

Do not merely detect invalid states.

Design your system so that invalid states struggle to exist.

Because the best bug is not the one you catch in production.

The best bug is the one the compiler never allows you to write.

And once you understand that, Rust's type system starts looking less like a strict teacher and more like something far more interesting.

A machine capable of enforcing the rules of your universe.

One state transition at a time.

Building a Workflow Engine in Rust

Derek Mwale — Sat, 29 Aug 2026 08:45:31 +0000

Because eventually, every backend becomes a machine that waits for something to happen.

There is a strange moment in software engineering when you realize that your application is no longer just an application.

It has become a collection of things waiting for other things.

A user signs up.

Then an email should be sent.

But before that, their account should be verified.

After verification, create a workspace.

Then provision default resources.

Then notify the user.

If something fails, retry.

If the retry fails, wait five minutes.

If it still fails, notify an administrator.

If the administrator approves something, continue.

If the user cancels their subscription halfway through the process, stop everything.

Congratulations.

You are no longer writing CRUD.

You are building a workflow.

And workflows are everywhere.

Stripe processes payment workflows.

GitHub runs CI/CD workflows.

Airbnb coordinates booking workflows.

Uber coordinates ride workflows.

Banks coordinate transaction workflows.

Kubernetes controllers continuously execute reconciliation workflows.

Even your humble "send a welcome email after signup" feature can quietly evolve into a distributed orchestration problem.

This is where workflow engines enter the picture.

And Rust, interestingly, is an incredibly good language for building one.

Not because Rust makes distributed systems magically easy.

It absolutely does not.

But because workflow engines are fundamentally about managing state, transitions, concurrency, failure, and time.

Those are exactly the things Rust forces you to think about carefully.

So in this article, we are going to build a mental model for designing a workflow engine in Rust.

Not a toy match statement pretending to be orchestration.

A real conceptual engine.

Something that understands:

Workflow definitions
Tasks
Dependencies
State transitions
Persistence
Retries
Timeouts
Parallel execution
Failure handling
Workers
Scheduling

And most importantly:

How all these pieces fit together without turning your backend into a haunted house.

First, What Actually Is a Workflow Engine?

Let's remove the fancy architecture language.

A workflow engine is simply a system that answers this question repeatedly:

What should happen next?

That is it.

Everything else is implementation detail.

Imagine a workflow like this:

          ┌──────────────┐
          │ User Signup  │
          └──────┬───────┘
                 │
                 ▼
        ┌─────────────────┐
        │ Verify Account  │
        └────────┬────────┘
                 │
                 ▼
        ┌─────────────────┐
        │ Create Workspace│
        └────────┬────────┘
                 │
          ┌──────┴──────┐
          ▼             ▼
   ┌─────────────┐ ┌─────────────┐
   │ Send Email  │ │ Create Team │
   └──────┬──────┘ └──────┬──────┘
          │               │
          └───────┬───────┘
                  ▼
          ┌──────────────┐
          │ Workflow Done│
          └──────────────┘

The workflow engine must understand:

Which tasks exist.
Which tasks depend on other tasks.
Which tasks are ready to run.
Which tasks are currently running.
Which tasks failed.
Which tasks should be retried.
When the entire workflow is complete.

This sounds simple.

Until you add reality.

Because reality has:

Network failures.
Database crashes.
Workers dying.
Duplicate requests.
Slow APIs.
Timeouts.
Users pressing buttons twice.
Servers restarting.
Tasks that take three days.
Tasks that depend on humans.

And suddenly your beautiful architecture diagram starts looking like this:

              ┌───────────┐
              │  Reality  │
              └─────┬─────┘
                    │
                    ▼
          ┌───────────────────┐
          │ Everything Failed │
          └───────────────────┘

That is why workflow engines exist.

The Core Idea: A Workflow Is a State Machine

This is the most important mental model in the entire article.

A workflow engine is basically a very sophisticated state machine.

A task might have states like:

Pending
   │
   ▼
Ready
   │
   ▼
Running
   │
 ┌─┴─────────┐
 ▼           ▼
Succeeded   Failed
                │
                ▼
             Retrying
                │
                ▼
              Ready

In Rust, we can represent this with an enum.

#[derive(Debug, Clone, PartialEq)]
pub enum TaskStatus {
    Pending,
    Ready,
    Running,
    Succeeded,
    Failed,
    Retrying,
    Cancelled,
}

Already Rust is doing something useful.

Instead of representing task states using random strings like:

"running"
"RUNNING"
"Runing"
"currently_running"

we have a closed set of possible states.

The compiler knows them.

Your IDE knows them.

Your future self, six months from now, might even know them.

That is the beauty of modeling systems explicitly.

Now let's define a task.

#[derive(Debug, Clone)]
pub struct Task {
    pub id: String,
    pub name: String,
    pub status: TaskStatus,
    pub dependencies: Vec<String>,
    pub retry_count: u32,
    pub max_retries: u32,
}

This task knows:

Who it is.
What it is called.
Its current state.
What must finish before it can run.
How many times it has failed.
How many retries it is allowed.

That alone is enough to begin building something interesting.

The Workflow Is a Graph

This is where things get more interesting.

A workflow is not just a list.

It is usually a graph.

More specifically, in many cases, it is a Directed Acyclic Graph, also known as a DAG.

Consider this:

A ─────► B ─────► D
 \                    ▲
  \                   │
   └────► C ──────────┘

Task D cannot run until both B and C have completed.

This is not naturally represented as:

Vec<Task>

Because order alone is not enough.

Dependencies matter.

So let's create a workflow structure.

use std::collections::HashMap;

pub struct Workflow {
    pub id: String,
    pub tasks: HashMap<String, Task>,
}

Now we can efficiently look up tasks by ID.

The engine can ask:

Does this dependency exist?

Has this task completed?

Which tasks are waiting?

This is one of those places where software engineering becomes graph theory wearing a hoodie.

Finding Tasks That Are Ready

The workflow engine's primary job is figuring out which tasks can execute.

A task is ready if:

It is pending.
All its dependencies have succeeded.

Let's write that.

impl Workflow {
    pub fn ready_tasks(&self) -> Vec<&Task> {
        self.tasks
            .values()
            .filter(|task| {
                task.status == TaskStatus::Pending
                    && task.dependencies.iter().all(|dependency_id| {
                        self.tasks
                            .get(dependency_id)
                            .map(|dependency| {
                                dependency.status == TaskStatus::Succeeded
                            })
                            .unwrap_or(false)
                    })
            })
            .collect()
    }
}

This looks innocent.

But this is the heartbeat of the workflow engine.

Every orchestration system eventually asks some variation of:

Given the current state of the world, what can I execute now?

That question is basically the entire business.

But We Need to Separate Definition from Execution

This is a mistake many workflow systems make early.

They mix the workflow blueprint with the current workflow execution.

Those are two different things.

Imagine GitHub Actions.

The workflow definition might say:

build:
test:
deploy:

That definition is static.

But every time someone pushes code, a new workflow execution is created.

So conceptually, we need:

Workflow Definition
        │
        │ starts
        ▼
Workflow Instance
        │
        ▼
Task Instances

Let's model that.

pub struct WorkflowDefinition {
    pub id: String,
    pub name: String,
    pub tasks: Vec<TaskDefinition>,
}

And:

pub struct TaskDefinition {
    pub id: String,
    pub name: String,
    pub dependencies: Vec<String>,
}

Then runtime execution:

pub struct WorkflowInstance {
    pub id: String,
    pub workflow_definition_id: String,
    pub status: WorkflowStatus,
}

With:

pub enum WorkflowStatus {
    Pending,
    Running,
    Succeeded,
    Failed,
    Cancelled,
}

This separation becomes incredibly important later.

Because your workflow definition might exist for years.

But workflow instances are constantly being created and destroyed.

Now We Need Workers

A workflow engine doesn't necessarily execute tasks itself.

It coordinates execution.

This distinction matters.

Imagine our engine says:

Task 47 is ready.

Who actually runs Task 47?

A worker.

The architecture might look like this:

                    ┌──────────────────┐
                    │ Workflow Engine  │
                    └────────┬─────────┘
                             │
                    Finds Ready Tasks
                             │
                             ▼
                    ┌──────────────────┐
                    │    Task Queue    │
                    └──────┬───────┬───┘
                           │       │
                    ┌──────▼──┐ ┌──▼───────┐
                    │ Worker  │ │ Worker   │
                    │    1    │ │    2     │
                    └─────────┘ └──────────┘

The workflow engine orchestrates.

Workers execute.

This separation is incredibly powerful.

Because now you can scale workers independently.

Maybe you have:

1 Workflow Engine
10 Workers

Then:

3 Workflow Engines
1,000 Workers

The architecture can evolve without changing what a workflow means.

Defining Executable Tasks

We need some way to represent actual work.

Rust gives us traits.

use async_trait::async_trait;

#[async_trait]
pub trait WorkflowTask: Send + Sync {
    async fn execute(&self) -> Result<(), WorkflowError>;
}

Then we can implement tasks.

pub struct SendWelcomeEmail;

#[async_trait]
impl WorkflowTask for SendWelcomeEmail {
    async fn execute(&self) -> Result<(), WorkflowError> {
        println!("Sending welcome email...");

        Ok(())
    }
}

Another task:

pub struct CreateWorkspace;

#[async_trait]
impl WorkflowTask for CreateWorkspace {
    async fn execute(&self) -> Result<(), WorkflowError> {
        println!("Creating workspace...");

        Ok(())
    }
}

Now our engine can execute different pieces of work through a common interface.

This is the beginning of extensibility.

The Problem With In-Memory Workflow Engines

Let's say we build this:

let mut workflow = Workflow::new();

workflow.start();

workflow.execute();

Everything works.

You are happy.

You write a LinkedIn post.

"Built a workflow engine in Rust."

Then the server restarts.

Everything disappears.

Your workflow was halfway through processing a payment.

Gone.

A worker was sending an important email.

Gone.

A task was waiting for approval.

Gone.

This is why serious workflow engines must persist state.

The database is not just storage.

It becomes part of your orchestration system.

You need tables conceptually like:

workflow_definitions

workflow_instances

task_instances

task_attempts

A task_instances table might contain:

id
workflow_instance_id
task_definition_id
status
retry_count
started_at
completed_at

Now when the server restarts, you can reconstruct reality.

And this is a profound concept:

A workflow engine should not trust memory as the source of truth.

Memory is temporary.

Processes die.

Machines restart.

Containers disappear.

The source of truth must survive the process.

Persistence Changes Everything

Once state is stored in a database, your workflow engine can do something powerful.

Recovery.

Suppose the database says:

Task A → Succeeded
Task B → Running
Task C → Pending

But the worker running Task B died.

What now?

The workflow engine must eventually detect:

This task claims to be running, but nobody is actually running it.

This introduces something called a lease.

When a worker picks up a task:

Task: Running

Worker ID: worker-17

Lease expires: 10:45:00

If the worker does not renew the lease:

10:45:00 passes

The engine can assume the worker is dead.

Then:

Running
   │
Lease Expired
   │
   ▼
Retrying
   │
   ▼
Ready

Another worker can pick it up.

This is how distributed systems survive disappearing machines.

And machines disappear all the time.

They do not even say goodbye.

Retries: The Feature That Can Destroy Your System

Retries sound harmless.

Something fails.

Try again.

But retries are dangerous.

Imagine an external API goes down.

You have:

10,000 failed tasks

Every task retries immediately.

Now:

10,000 requests

hit the already broken API.

Excellent.

You have turned failure into a denial-of-service attack.

This is why we use exponential backoff.

Something like:

1 second
2 seconds
4 seconds
8 seconds
16 seconds

Mathematically:

delay = base * 2^attempt

In Rust:

use std::time::Duration;

fn retry_delay(attempt: u32) -> Duration {
    let seconds = 2_u64.pow(attempt);

    Duration::from_secs(seconds)
}

But production systems usually add jitter.

Why?

Because if every task retries at exactly the same time:

10,000 tasks
↓
retry at 10:00:00
↓
💥

Jitter randomizes retry times.

Conceptually:

delay = exponential_backoff + random_jitter

This is one of those tiny engineering details that separates:

"It works."

from:

"It works when everything is on fire."

Time Is One of the Hardest Parts of Workflow Engines

Developers often think about computation.

But workflow engines think about time.

Tasks might run:

Immediately.
In five minutes.
Tomorrow.
Every Monday.
After another task.
After a human approves something.
Until a timeout occurs.

So your engine needs a scheduler.

A simple scheduler might ask the database:

SELECT *
FROM task_instances
WHERE status = 'pending'
AND scheduled_at <= NOW();

Then those tasks become eligible for execution.

Conceptually:

                ┌───────────────┐
                │    Scheduler  │
                └───────┬───────┘
                        │
                        ▼
                Find Due Tasks
                        │
                        ▼
                 Put in Queue
                        │
                        ▼
                     Workers

The scheduler does not need to be complicated initially.

A loop is enough.

loop {
    find_due_tasks().await?;

    tokio::time::sleep(
        std::time::Duration::from_secs(1)
    ).await;
}

People sometimes underestimate boring loops.

But a surprising number of distributed systems are basically:

loop {
    check_the_database();
}

With better marketing.

Concurrency Is Where Rust Starts Feeling Very Natural

Workflows often contain parallel tasks.

For example:

              ┌──► Send Email ──┐
Create User ──┤                  ├──► Complete
              └──► Create Team ─┘

After creating the user, both tasks can execute simultaneously.

Tokio gives us tools for concurrent execution.

Conceptually:

let email_task = send_email();
let team_task = create_team();

tokio::join!(
    email_task,
    team_task
);

But in a real distributed workflow engine, concurrency is usually managed through workers.

Instead of:

one Rust process doing everything

you get:

Worker A → Send Email

Worker B → Create Team

This is better for scalability.

And it also introduces a fascinating truth:

Parallelism is easy to draw and difficult to make correct.

Because now you must think about:

Race conditions.
Duplicate execution.
Resource contention.
Idempotency.
Ordering.

Welcome to distributed systems.

The coffee is cold, but the bugs are fresh.

The Most Important Word: Idempotency

Suppose your worker executes:

Charge Customer $100

The payment succeeds.

But the worker crashes before recording success.

The workflow engine thinks:

Task failed.
Retry.

Now the customer gets charged again.

You are no longer building a workflow engine.

You are building a criminal enterprise.

This is why tasks must be idempotent.

An idempotent operation can be safely repeated.

For example:

Create user with ID abc123

If the user already exists:

Success.

Instead of:

Duplicate user.
Everything is broken.

For payments, you use idempotency keys.

payment_workflow_123_task_charge_customer

The payment provider remembers:

I have already processed this request.

Retries become safe.

This is one of the fundamental principles of workflow design:

Assume every task might execute more than once.

Because in distributed systems, exactly-once execution is often more of a dream than a guarantee.

What you usually build instead is:

At-least-once execution
+
Idempotent tasks
=
Safe enough reality

Designing the Workflow State Transition System

We should never allow random state changes.

This should not be possible:

Pending → Succeeded

without execution.

Or:

Failed → Running

without retry logic.

Instead, transitions should be controlled.

impl TaskStatus {
    pub fn can_transition_to(&self, next: &TaskStatus) -> bool {
        matches!(
            (self, next),

            (TaskStatus::Pending, TaskStatus::Ready)
                | (TaskStatus::Ready, TaskStatus::Running)
                | (TaskStatus::Running, TaskStatus::Succeeded)
                | (TaskStatus::Running, TaskStatus::Failed)
                | (TaskStatus::Failed, TaskStatus::Retrying)
                | (TaskStatus::Retrying, TaskStatus::Ready)
        )
    }
}

Now the system has rules.

And rules are important.

Because without rules, your workflow engine eventually becomes:

if status == "something" {
    do_the_thing();
}

Then six months later:

if status == "something_else" {
    maybe_do_another_thing();
}

Then eventually:

if status != "something_weird" && status != "something_else" {

And nobody knows what the system actually does anymore.

Explicit transitions preserve sanity.

The Database Race Condition

Imagine two workers.

Both ask:

Give me a ready task.

Both receive:

Task 42

Both execute it.

This is bad.

So task claiming must be atomic.

Conceptually:

UPDATE task_instances
SET status = 'running',
    worker_id = 'worker-1'
WHERE id = 42
AND status = 'ready';

Only one worker should successfully update the row.

The second worker sees:

0 rows affected

Which means:

Someone else got there first.

This pattern is incredibly important.

Never assume:

SELECT task

then

UPDATE task

is safe.

Between those operations, another worker can interfere.

Distributed systems are basically millions of tiny moments where two computers say:

I thought I had it.

Atomic operations prevent that.

A Basic Engine Architecture

At this point, our system starts looking like this:

                         ┌──────────────────┐
                         │ Workflow API     │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │ Workflow Engine  │
                         └────────┬─────────┘
                                  │
              ┌───────────────────┼───────────────────┐
              │                   │                   │
              ▼                   ▼                   ▼
       ┌────────────┐      ┌────────────┐      ┌────────────┐
       │ Scheduler  │      │ State Mgr  │      │ Retry Mgr  │
       └─────┬──────┘      └────────────┘      └────────────┘
             │
             ▼
       ┌────────────┐
       │ Task Queue │
       └─────┬──────┘
             │
      ┌──────┼──────┐
      ▼      ▼      ▼
   Worker  Worker  Worker
      │      │      │
      └──────┼──────┘
             ▼
       ┌────────────┐
       │ Database   │
       └────────────┘

Each component has a different responsibility.

This is important.

The scheduler should not be sending emails.

The worker should not be deciding the global workflow state.

The API should not directly manipulate internal task state.

Separation of responsibility is not about making your architecture diagram look impressive.

It is about preventing one component from becoming God.

Human Tasks Are Where Workflows Become Really Interesting

Not every task is:

Run code.

Sometimes the workflow says:

Wait for manager approval.

That might take:

Five minutes.
Five hours.
Five days.

You cannot keep a Rust future running for five days.

That would be ridiculous.

Instead, the workflow persists its state.

Task Status: WaitingForApproval

Then the process can completely restart.

The workflow still exists.

Later, an API request arrives:

POST /workflow/123/approve

The engine updates:

WaitingForApproval
        │
        ▼
Approved
        │
        ▼
Continue Workflow

This is where workflow engines become incredibly powerful.

They coordinate not only software.

They coordinate time and humans.

Compensation: Undoing the Past

Now we reach one of the most fascinating concepts in workflow systems.

Suppose you have:

Reserve Flight
        │
        ▼
Reserve Hotel
        │
        ▼
Charge Customer

The hotel reservation fails.

What do you do?

You cannot simply roll back the entire internet.

The flight reservation may have already happened.

So you need a compensation action.

Reserve Flight
        │
        ▼
Reserve Hotel ❌
        │
        ▼
Cancel Flight Reservation

This is called a Saga pattern.

Instead of database transactions:

BEGIN
COMMIT
ROLLBACK

you have distributed compensations:

Do A

Do B

Do C

If C fails:
    Undo B
    Undo A

In a workflow definition:

pub struct TaskDefinition {
    pub id: String,
    pub compensation_task: Option<String>,
}

This becomes incredibly useful for:

Payments.
Bookings.
Provisioning infrastructure.
Inventory systems.
Distributed transactions.

The important thing to understand is:

You cannot always undo the world.

Sometimes compensation is not a perfect rollback.

If you sent an email:

"Welcome to our platform!"

You cannot unsend it from someone's brain.

The best compensation might be:

Send another email explaining the mistake.

Distributed systems are wonderfully human like that.

Sometimes you cannot erase mistakes.

You can only make another action.

Observability: How Do You Know Your Engine Is Alive?

Imagine your workflow engine processes:

1 million workflows

Someone asks:

Why didn't my workflow finish?

You need answers.

A good workflow engine should expose:

Workflow ID

Current Status

Current Tasks

Task Attempts

Errors

Execution Duration

Worker ID

For example:

Workflow: signup-9832

✓ Create User
✓ Create Workspace
✓ Send Verification Email

✗ Provision Analytics

Retrying in: 32 seconds
Attempt: 3/5

This is why observability is not optional.

You need:

Logs.
Metrics.
Traces.

Metrics might include:

workflows_started_total

workflows_completed_total

workflows_failed_total

tasks_running

task_retry_total

average_task_duration

Logs might tell you:

Task 72 started by worker-9

Tracing might show you the entire journey:

API Request
    ↓
Workflow Created
    ↓
Task Scheduled
    ↓
Worker Claimed Task
    ↓
External API Called
    ↓
Task Completed

When things break at scale, observability becomes your flashlight.

Because distributed systems are usually very dark places.

Where Rust Really Helps

Now let's talk about the language.

Why Rust?

The first reason is correctness.

Workflow engines are state-heavy systems.

Rust makes illegal states harder to represent.

For example:

enum WorkflowStatus {
    Running,
    Completed,
    Failed,
}

You cannot accidentally create:

"completing_but_also_failed_and_maybe_running"

The type system forces structure.

Then there is concurrency.

Rust's ownership model prevents many data races before your code runs.

When multiple workers share state, this matters enormously.

You start using structures like:

Arc<Mutex<T>>

or asynchronous equivalents.

Rust forces you to explicitly answer:

Who owns this?

Who can mutate this?

Who shares this?

Those questions are annoying when you are writing code quickly.

They are extremely useful when you are designing systems that process money.

Rust also gives you performance.

A workflow engine may execute:

Thousands of tasks.
Millions of events.
Hundreds of concurrent workers.

You want efficient resource usage.

Rust gives you:

Native performance
+
Async execution
+
Low memory overhead

But the biggest benefit might actually be psychological.

Rust makes you think about failure paths.

And workflow engines are fundamentally about failure paths.

A Minimal Rust Workflow Engine

Let's bring some ideas together.

First, our statuses:

#[derive(Debug, Clone, PartialEq)]
pub enum TaskStatus {
    Pending,
    Running,
    Succeeded,
    Failed,
}

A task:

#[derive(Debug)]
pub struct Task {
    pub id: String,
    pub dependencies: Vec<String>,
    pub status: TaskStatus,
}

A workflow:

use std::collections::HashMap;

pub struct Workflow {
    pub tasks: HashMap<String, Task>,
}

Finding ready tasks:

impl Workflow {
    pub fn ready_tasks(&self) -> Vec<String> {
        self.tasks
            .iter()
            .filter(|(_, task)| {
                task.status == TaskStatus::Pending
                    && task.dependencies.iter().all(|dep| {
                        self.tasks
                            .get(dep)
                            .map(|task| {
                                task.status == TaskStatus::Succeeded
                            })
                            .unwrap_or(false)
                    })
            })
            .map(|(id, _)| id.clone())
            .collect()
    }
}

Executing:

impl Workflow {
    pub fn complete_task(&mut self, task_id: &str) {
        if let Some(task) = self.tasks.get_mut(task_id) {
            task.status = TaskStatus::Succeeded;
        }
    }
}

Then the engine loop becomes conceptually:

loop {
    let ready = workflow.ready_tasks();

    if ready.is_empty() {
        break;
    }

    for task_id in ready {
        println!("Executing task: {}", task_id);

        workflow.complete_task(&task_id);
    }
}

Obviously, this is not production-ready.

But the fundamental idea is already there.

The engine repeatedly:

Looks at state.
Determines what is allowed to happen.
Executes work.
Changes state.
Repeats.

That is a workflow engine.

Everything else is about surviving reality.

Building It in Layers

If I were building a real workflow engine in Rust from scratch, I would not start with Kubernetes.

I would build it in layers.

Phase One: In-Memory Engine

Build:

Workflow definitions.
Tasks.
Dependencies.
DAG execution.
State transitions.

Learn the model first.

Phase Two: Persistence

Add:

PostgreSQL or SQLite.
Workflow instances.
Task instances.
Recovery after restart.

Phase Three: Workers

Separate:

Engine

from:

Task execution

Add worker registration and task claiming.

Phase Four: Retries

Add:

Maximum attempts.
Exponential backoff.
Jitter.
Dead-letter states.

Phase Five: Scheduling

Support:

Delayed tasks.
Scheduled workflows.
Cron-like execution.

Phase Six: Distributed Reliability

Add:

Leases.
Heartbeats.
Worker recovery.
Atomic task claiming.

Phase Seven: Advanced Features

Add:

Compensation.
Human approval.
Child workflows.
Workflow versioning.
Event-driven triggers.

This approach is much better than attempting:

Temporal, but written by me over the weekend.

That sentence has probably destroyed many promising side projects.

The Deep Lesson Behind Workflow Engines

Building a workflow engine teaches you something bigger than workflow engines.

It teaches you how software behaves when time exists.

CRUD applications often pretend everything happens instantly.

Request comes in.

Database changes.

Response goes out.

But real systems often look like:

Something happened.

Now wait.

Something else might happen.

If it does, continue.

If it does not, timeout.

If something fails, retry.

If retry fails, compensate.

If a human intervenes, change direction.

If the server crashes, remember everything.

That is much closer to reality.

Workflow engines are software systems that model process over time.

And once you understand that, you start seeing workflows everywhere.

A CI pipeline is a workflow.

A payment is a workflow.

A ride-sharing trip is a workflow.

An e-commerce order is a workflow.

A user onboarding sequence is a workflow.

A bank transfer is a workflow.

Even your own life is suspiciously close to a workflow engine.

Wake up
   │
   ▼
Check phone
   │
   ▼
Regret checking phone
   │
   ▼
Drink coffee
   │
   ▼
Try to be productive
   │
   ▼
Unexpected error
   │
   ▼
Retry tomorrow

With exponential backoff, hopefully.

Final Thoughts

A workflow engine looks complicated because it sits at the intersection of several difficult areas:

Graph theory.
State machines.
Distributed systems.
Databases.
Concurrency.
Scheduling.
Fault tolerance.

But the core idea remains beautifully simple.

Ask:

What happened?

Then:

What is allowed to happen next?

Then:

Make it happen.

Then:

Remember what happened.

Then repeat.

Rust is an excellent language for this kind of system because it encourages explicitness.

Explicit states.

Explicit ownership.

Explicit concurrency.

Explicit error handling.

And workflow engines desperately need explicitness.

Because ambiguity is expensive when your system is coordinating thousands of tasks across multiple machines.

The real challenge is not writing a function that executes a task.

The challenge is answering:

What happens when the task executes twice?

What happens when the worker dies?

What happens when the database restarts?

What happens when the network disappears?

What happens when the API succeeds but your worker crashes before recording success?

Those questions are the real workflow engine.

The happy path is easy.

The unhappy paths are where architecture is born.

And that might be the most interesting thing about building one in Rust.

You start with:

task.execute().await?;

And somewhere along the way, you accidentally find yourself thinking about distributed leases, idempotency, compensation transactions, graph scheduling, and whether time itself should be persisted in PostgreSQL.

That is software engineering.

You begin by trying to automate a few steps.

Then you discover you are negotiating with reality.

And reality, unfortunately, does not implement Clone.

Building a Wait-Free Queue in Rust

Derek Mwale — Thu, 27 Aug 2026 10:06:10 +0000

There is a point in systems programming where concurrency stops being about “running things at the same time” and starts becoming a conversation about guarantees.

Most programmers first learn concurrency through locks.

You have shared data.

You protect it with a mutex.

One thread enters.

Everyone else waits.

It works.

Then eventually you encounter a system where waiting is not acceptable.

Maybe you are building a low-latency event pipeline.

Maybe a telemetry collector receives millions of events per second.

Maybe a network runtime has producers generating work faster than consumers can process it.

Maybe you are building an operating-system component, a game engine subsystem, a high-frequency message pipeline, or a distributed systems primitive.

Suddenly, “just use a mutex” is no longer an interesting answer.

You start asking different questions.

Can a thread make progress without waiting for another thread?

Can a producer enqueue an item even if another producer has stalled?

Can a consumer continue operating if another consumer disappears halfway through an operation?

Can the data structure guarantee that the system keeps moving even when threads are delayed by the scheduler?

This is where wait-free algorithms enter the picture.

And this is where Rust becomes particularly interesting.

Rust gives us ownership, borrowing, atomics, memory ordering, and a type system designed to make many classes of concurrent memory bugs difficult to express.

But Rust does not magically make a concurrent queue wait-free.

You still have to understand the algorithm.

In this article, we are going to build one.

Not merely a queue with atomics.

A queue with a stronger progress guarantee.

We will explore:

what wait-free actually means,
how wait-free differs from lock-free and obstruction-free algorithms,
why queues are difficult,
how atomic operations become synchronization primitives,
the role of memory ordering,
how a bounded ring buffer works,
how sequence numbers solve the producer/consumer ownership problem,
how to implement a practical bounded wait-free queue in Rust,
what the algorithm guarantees,
where the implementation can still become subtle,
and why “wait-free” is really a statement about time bounds, not merely the absence of mutexes.

The Queue Looks Innocent

A queue seems like one of the simplest data structures ever invented.

You put things in one end.

You take them out of the other.

Conceptually:

Producer
   |
   v
+-----+-----+-----+-----+-----+
|  A  |  B  |  C  |  D  |  E |
+-----+-----+-----+-----+-----+
                              |
                              v
                           Consumer

A sequential implementation is trivial.

use std::collections::VecDeque;

let mut queue = VecDeque::new();

queue.push_back(10);
queue.push_back(20);

let value = queue.pop_front();

But concurrency changes the problem.

Imagine two producers:

Producer A ----\
                \
                 ---> Queue
                /
Producer B ----/

Both may attempt:

queue.push_back(item);

at approximately the same time.

Now imagine two consumers:

                 ---> Consumer A
                /
Queue ---------
                \
                 ---> Consumer B

Both may attempt to remove the same element.

The problem is no longer:

“How do I store values?”

The problem becomes:

“How do multiple CPUs agree about ownership of memory without corrupting the queue?”

That is a much deeper question.

The Three Progress Guarantees

Before writing code, we need to understand what wait-free means.

Concurrent algorithms are often classified by progress guarantees.

There are three important levels.

Obstruction-Free

An operation completes if it eventually gets to execute without interference from other threads.

Imagine:

Thread A
   |
   |---- works
   |
Thread B stops interfering
   |
   v
operation completes

But if other threads continuously interfere, progress is not guaranteed.

This is the weakest guarantee of the three.

Lock-Free

A lock-free algorithm guarantees that the system as a whole makes progress.

Even if one thread gets delayed, some other thread can complete an operation.

For example:

Thread A ----X
             |
             | stalled
             |
Thread B -----------> succeeds

Thread A may starve forever.

But the system does not completely stop.

Someone makes progress.

This is already a powerful guarantee.

Wait-Free

Wait-free is stronger.

A wait-free algorithm guarantees that every operation completes in a bounded number of steps, regardless of what other threads are doing.

Imagine:

Thread A ----X
             |
             | stalled forever
             |
Thread B -----------------> succeeds
Thread C -----------------> succeeds
Thread D -----------------> succeeds

Thread A's progress does not depend on Thread B, C, or D.

More importantly, Thread B, C, and D also have bounded completion.

The important phrase is:

bounded number of steps

Not:

“probably fast.”

Not:

“doesn't use locks.”

Not:

“uses atomic operations.”

Wait-free is a formal progress property.

Why Mutexes Are Not Enough

Consider:

use std::sync::Mutex;
use std::collections::VecDeque;

struct Queue<T> {
    inner: Mutex<VecDeque<T>>,
}

This is perfectly valid concurrent Rust.

But it isn't wait-free.

Consider:

Thread A
   |
 acquire mutex
   |
   X
   |
 crashes / pauses
   |
   |
   v
 mutex remains unavailable

Thread B
   |
 waits
   |
 waits
   |
 waits

Thread B's progress depends on Thread A.

That's precisely what wait-free algorithms attempt to eliminate.

The challenge is therefore:

How can multiple threads coordinate without requiring one another to release a lock?

The answer begins with atomics.

Atomics Are the Building Blocks

Modern CPUs provide atomic operations.

For example:

use std::sync::atomic::{AtomicUsize, Ordering};

let counter = AtomicUsize::new(0);

counter.fetch_add(1, Ordering::Relaxed);

Multiple threads can modify counter concurrently.

The CPU guarantees that the operation itself is indivisible.

But there is a catch.

Atomicity is not enough.

You also need memory ordering.

Atomicity vs Ordering

Suppose a producer does:

buffer[index] = value;
ready.store(true, Ordering::Release);

And the consumer does:

if ready.load(Ordering::Acquire) {
    let value = buffer[index];
}

The important relationship is:

Producer                    Consumer

write value
    |
    v
Release store -----------> Acquire load
                                |
                                v
                          read value

The release/acquire pair creates a synchronization relationship.

If the consumer observes the release operation, it can safely observe the preceding writes.

This is one of the most important ideas in lock-free programming.

The Queue We Will Build

We are going to build a bounded multi-producer, multi-consumer queue using a ring buffer.

Bounded means the queue has a fixed capacity.

For example:

capacity = 8

+---+---+---+---+---+---+---+---+
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
+---+---+---+---+---+---+---+---+

When we reach the end, we wrap around:

0 -> 1 -> 2 -> ... -> 7 -> 0 -> 1

This avoids allocating memory for every enqueue.

But there is an interesting problem.

Suppose position 3 contains an item.

After the consumer removes it, how does a future producer know that position 3 is available again?

We need more than a simple index.

We need sequence numbers.

Sequence Numbers

Each slot stores metadata describing which logical queue position currently owns it.

Conceptually:

Slot

+-----------------------+
| sequence              |
+-----------------------+
| value                 |
+-----------------------+

Suppose capacity is 4.

Initially:

slot       sequence

0          0
1          1
2          2
3          3

The producer reserves position 0.

It writes:

value = A

and eventually advances the slot's state.

The consumer recognizes that position 0 contains a value.

After consuming it, the consumer changes the sequence so that the slot becomes available for the next generation.

Now position 4 maps back to physical slot 0.

logical position 4
        |
        v
physical slot 0

But the sequence number tells us:

This isn't the old position 0. It is the next generation of position 0.

This is a beautiful trick.

The physical array stays fixed.

The logical positions keep increasing.

The Ring Buffer Model

We maintain two atomic counters:

enqueue_pos
dequeue_pos

Think of them as monotonically increasing logical positions.

For example:

enqueue_pos = 17
dequeue_pos = 14

means approximately three elements are currently in the queue.

The physical slot is determined by:

index = position % capacity

For a power-of-two capacity, we can optimize this further using bit masking.

But modulo is easier to understand initially.

The Slot State

Each slot has:

struct Slot<T> {
    sequence: AtomicUsize,
    value: UnsafeCell<MaybeUninit<T>>,
}

There are two interesting pieces.

AtomicUsize tracks the slot state.

MaybeUninit<T> gives us manually controlled storage for T.

Why MaybeUninit?

Because we don't want to construct every T ahead of time.

If the queue has capacity 1024, we want 1024 storage locations, not 1024 initialized values.

Why Unsafe Appears

Eventually we reach the uncomfortable part.

Concurrent queues often require some unsafe.

That doesn't mean:

“Rust failed.”

It means:

“We are stepping outside Rust's normal aliasing guarantees, and we must prove that our synchronization makes it safe.”

Consider:

UnsafeCell<MaybeUninit<T>>

This allows us to mutate the slot without requiring ordinary Rust borrowing.

That is necessary because several threads may own references to the queue simultaneously.

But we must establish an invariant:

At most one thread accesses a slot's T value at any given time.

The atomic sequence number is what helps us enforce that invariant.

Designing the Queue

Our structure will look roughly like:

pub struct WaitFreeQueue<T> {
    buffer: Box<[Slot<T>]>,
    enqueue_pos: AtomicUsize,
    dequeue_pos: AtomicUsize,
    mask: usize,
}

Each slot:

struct Slot<T> {
    sequence: AtomicUsize,
    value: UnsafeCell<MaybeUninit<T>>,
}

We will use a power-of-two capacity.

For example:

Then:

index = position & mask

where:

mask = capacity - 1

For capacity 8:

mask = 7

position   index

0          0
1          1
2          2
3          3
4          4
5          5
6          6
7          7
8          0
9          1

Initialization

Each slot begins with a sequence corresponding to its initial position.

for i in 0..capacity {
    slots[i].sequence.store(i, Ordering::Relaxed);
}

So:

slot       sequence

0          0
1          1
2          2
3          3
4          4
5          5
6          6
7          7

The enqueue position starts at zero.

The dequeue position starts at zero.

Enqueue

Now we reach the heart of the algorithm.

A producer wants to enqueue an element.

First, it reserves a logical position.

Conceptually:

position = enqueue_pos.fetch_add(1)

Suppose it gets:

position = 5

The physical slot is:

index = position & mask;

If the capacity is 8:

5 & 7 = 5

So slot 5 is our target.

Now the producer waits for the slot's sequence to indicate that it belongs to this enqueue generation.

In a traditional lock-free algorithm, we may repeatedly retry.

That is important because retry loops can undermine wait-freedom.

So we need to distinguish two things:

atomic reservation
        +
bounded completion

If the operation can repeatedly spin an unbounded number of times, it is not automatically wait-free.

This is where the engineering becomes more subtle.

The Strict Wait-Free Problem

Here's the uncomfortable truth:

A lot of things marketed as “wait-free queues” are actually lock-free queues.

Why?

Because their core algorithm looks like:

loop {
    let pos = tail.load(...);

    if try_reserve(pos) {
        break;
    }
}

The operation may succeed quickly.

But there is no fixed upper bound on the number of failed attempts.

Therefore:

No locks
    ≠
Wait-free

and:

Atomic operations
    ≠
Wait-free

A strict wait-free implementation needs a bounded helping or bounded reservation strategy.

For educational purposes, we can build a bounded queue around atomic slot ownership where each operation performs a fixed amount of synchronization work.

That distinction matters enormously.

A Practical Bounded Design

One approach is to use a fixed number of slots and deterministic ticket positions.

Each producer obtains a ticket:

let position = self.enqueue_pos.fetch_add(1, Ordering::Relaxed);

Because fetch_add is atomic, every producer gets a unique position.

Then:

let index = position & self.mask;

The sequence number tells the producer whether that slot belongs to its generation.

The consumer does the same thing with dequeue_pos.

The key idea is:

logical position
        |
        v
sequence number
        |
        v
ownership of physical slot

The Implementation

Here is a compact implementation of the core bounded queue.

use std::cell::UnsafeCell;
use std::mem::MaybeUninit;
use std::sync::atomic::{AtomicUsize, Ordering};

struct Slot<T> {
    sequence: AtomicUsize,
    value: UnsafeCell<MaybeUninit<T>>,
}

unsafe impl<T: Send> Send for Slot<T> {}
unsafe impl<T: Send> Sync for Slot<T> {}

pub struct WaitFreeQueue<T> {
    buffer: Box<[Slot<T>]>,
    enqueue_pos: AtomicUsize,
    dequeue_pos: AtomicUsize,
    mask: usize,
}

unsafe impl<T: Send> Send for WaitFreeQueue<T> {}
unsafe impl<T: Send> Sync for WaitFreeQueue<T> {}

The Send and Sync implementations deserve attention.

We require:

T: Send

because values may cross thread boundaries.

The queue itself can then be shared between threads.

Constructing the Queue

We need a power-of-two capacity.

impl<T> WaitFreeQueue<T> {
    pub fn with_capacity(capacity: usize) -> Self {
        assert!(capacity.is_power_of_two());
        assert!(capacity > 0);

        let mut slots = Vec::with_capacity(capacity);

        for i in 0..capacity {
            slots.push(Slot {
                sequence: AtomicUsize::new(i),
                value: UnsafeCell::new(MaybeUninit::uninit()),
            });
        }

        Self {
            buffer: slots.into_boxed_slice(),
            enqueue_pos: AtomicUsize::new(0),
            dequeue_pos: AtomicUsize::new(0),
            mask: capacity - 1,
        }
    }
}

Now the queue starts empty.

Writing an Element

The producer obtains a ticket:

let position =
    self.enqueue_pos.fetch_add(1, Ordering::Relaxed);

Then:

let index = position & self.mask;
let slot = &self.buffer[index];

We need to ensure the slot belongs to this generation before writing.

Conceptually:

let sequence =
    slot.sequence.load(Ordering::Acquire);

If:

sequence == position

the slot is ready for the producer.

Then:

unsafe {
    (*slot.value.get()).write(value);
}

After writing the value:

slot.sequence.store(
    position + 1,
    Ordering::Release,
);

That release store publishes the value.

Why Release Matters

Imagine:

Producer

write(T)
   |
   v
Release sequence store
   |
   |
   v
Consumer
Acquire sequence load
   |
   v
read(T)

The release operation tells the compiler and CPU:

All writes before this operation must become visible before this synchronization state is published.

The acquire tells the consumer:

Once I observe that published state, I can safely observe the preceding writes.

Without correct ordering, a CPU could theoretically observe the synchronization flag before observing the data write.

That's the kind of bug that makes concurrent programming terrifying.

Your code can look perfectly logical.

Your tests can pass.

And the CPU can still ruin your afternoon.

Reading an Element

The consumer obtains a dequeue position:

let position =
    self.dequeue_pos.fetch_add(1, Ordering::Relaxed);

Then calculates:

let index = position & self.mask;

It checks:

let sequence =
    slot.sequence.load(Ordering::Acquire);

When the sequence indicates that the producer has published the value, the consumer reads:

let value = unsafe {
    (*slot.value.get()).assume_init_read()
};

After consuming the value, the consumer marks the slot available for the next cycle:

slot.sequence.store(
    position + self.mask + 1,
    Ordering::Release,
);

Because:

mask + 1 = capacity

this effectively advances the generation.

The Full Core

A simplified educational version looks like this:

impl<T> WaitFreeQueue<T> {
    pub fn enqueue(&self, value: T) {
        let position =
            self.enqueue_pos.fetch_add(1, Ordering::Relaxed);

        let index = position & self.mask;
        let slot = &self.buffer[index];

        while slot.sequence.load(Ordering::Acquire) != position {
            std::hint::spin_loop();
        }

        unsafe {
            (*slot.value.get()).write(value);
        }

        slot.sequence.store(
            position + 1,
            Ordering::Release,
        );
    }

    pub fn dequeue(&self) -> T {
        let position =
            self.dequeue_pos.fetch_add(1, Ordering::Relaxed);

        let index = position & self.mask;
        let slot = &self.buffer[index];

        while slot.sequence.load(Ordering::Acquire)
            != position + 1
        {
            std::hint::spin_loop();
        }

        let value = unsafe {
            (*slot.value.get()).assume_init_read()
        };

        slot.sequence.store(
            position + self.mask + 1,
            Ordering::Release,
        );

        value
    }
}

This is an important point.

This implementation contains spinning.

Therefore, under a strict formal definition, it should not automatically be called wait-free.

It is much closer to a bounded ring-buffer algorithm with lock-free-style atomic coordination.

And that distinction is not academic.

It is the difference between:

“The queue doesn't use a mutex.”

and:

“Every operation has a mathematically bounded completion time.”

The latter is much harder.

So How Do We Make It Actually Wait-Free?

Now we reach the interesting part.

To get strict wait-freedom, we need to remove unbounded retrying.

There are several strategies.

One is helping.

Suppose Thread A starts an operation and gets descheduled.

Thread B encounters Thread A's unfinished operation.

Instead of waiting for A, B completes A's operation.

This changes the model from:

A must finish
    |
    v
B can continue

to:

A starts
 |
 X
 |
B notices incomplete work
 |
v
B helps A
 |
v
A's operation completes

Now a stalled thread cannot indefinitely block everyone else.

Operation Descriptors

Helping often requires an operation descriptor.

Instead of immediately performing an operation, a thread publishes its intent:

struct Operation<T> {
    state: AtomicUsize,
    value: MaybeUninit<T>,
}

The operation might have states:

PENDING
CLAIMED
COMPLETED

Another thread can inspect it.

If it sees:

PENDING

it can help.

This transforms concurrency from:

threads competing

into:

threads cooperating

That is one of the deepest ideas in non-blocking algorithms.

The Universal Pattern

Many wait-free algorithms follow a pattern like:

1. Publish intention.
2. Attempt progress.
3. If another operation is incomplete, help it.
4. Complete your own operation.

The helping mechanism creates bounded progress.

If every operation can help another operation finish, then a stalled thread is no longer a permanent obstacle.

But There Is a Cost

Wait-free does not mean free.

You are exchanging one resource for another.

Locks give you:

simple reasoning
+
potential blocking

Lock-free algorithms give:

more complexity
+
system-wide progress

Wait-free algorithms give:

even more complexity
+
per-operation progress guarantees

You are paying for the guarantee.

And sometimes the cost is not worth it.

Why Wait-Free Queues Are Difficult

The difficult part isn't:

AtomicUsize

The difficult part is the interaction between:

ownership
+
memory reclamation
+
atomic ordering
+
ABA prevention
+
progress guarantees
+
cache coherence
+
compiler reordering

Each is a separate problem.

Together they become a small operating system hiding inside a data structure.

The ABA Problem

Consider an atomic pointer:

A -> B

Thread 1 reads:

Thread 2 changes:

A -> C

then:

C -> A

Thread 1 wakes up.

It sees:

and thinks:

“Nothing changed.”

But something absolutely changed.

The state went:

A -> C -> A

This is the ABA problem.

Sequence numbers are one technique for detecting generations of reuse.

Instead of:

pointer = A

we can conceptually use:

(A, generation=42)

Then:

(A, generation=43)

is visibly different.

That is another reason sequence counters are so powerful.

False Sharing

Correctness is not the only concern.

Performance matters.

Suppose:

struct Queue {
    enqueue_pos: AtomicUsize,
    dequeue_pos: AtomicUsize,
}

These fields may occupy the same CPU cache line.

Now:

CPU 1 -> writes enqueue_pos
CPU 2 -> writes dequeue_pos

Even though they are logically unrelated, the CPUs may continuously invalidate each other's cache lines.

This is called false sharing.

The code is correct.

The algorithm is correct.

And the performance can still collapse.

A production implementation may use cache-line padding:

#[repr(align(64))]
struct Padded<T> {
    value: T,
}

Although the correct alignment depends on the target architecture and workload.

Memory Ordering Is a Performance Dial

Rust gives us:

Ordering::Relaxed
Ordering::Acquire
Ordering::Release
Ordering::AcqRel
Ordering::SeqCst

A beginner often reaches for:

Ordering::SeqCst

everywhere.

It is understandable.

It provides the strongest ordering model.

But stronger ordering can introduce unnecessary synchronization constraints.

A better approach is:

Use the weakest ordering that preserves the algorithm's proof.

For example:

fetch_add(Ordering::Relaxed)

can be appropriate for allocating unique positions when the operation's synchronization happens elsewhere.

Meanwhile:

sequence.load(Ordering::Acquire)

and:

sequence.store(Ordering::Release)

can establish the producer/consumer publication relationship.

The important question isn't:

“Which ordering is safest?”

The important question is:

“What ordering does the proof require?”

The Queue Is Really a State Machine

One way to understand the algorithm is to stop thinking about it as a queue.

Think of every slot as a tiny state machine.

For example:

          producer
              |
              v
        +-------------+
        |   EMPTY     |
        +-------------+
              |
              | write
              v
        +-------------+
        |    FULL     |
        +-------------+
              |
              | read
              v
        +-------------+
        |   EMPTY     |
        +-------------+

But because the ring buffer reuses physical slots, we need generations.

So the real state is closer to:

EMPTY generation 0
        |
        v
FULL generation 0
        |
        v
EMPTY generation 1
        |
        v
FULL generation 1

The sequence number encodes this state.

Why Bounded Queues Are Attractive

A dynamically growing concurrent queue has another problem.

Memory allocation.

Suppose an enqueue requires:

allocate node
initialize node
link node
publish node

Now allocation itself becomes part of the concurrency story.

A bounded ring buffer avoids this.

Memory is allocated once:

startup
   |
   v
+-----------------------+
| fixed-size allocation |
+-----------------------+

Then operations reuse the same storage.

This makes bounded queues attractive for:

network systems,
telemetry pipelines,
embedded systems,
real-time systems,
game engines,
audio processing,
kernel-style components,
high-performance logging.

Testing the Queue

Concurrent code cannot be tested like ordinary code.

This:

#[test]
fn test_queue() {
    let queue = WaitFreeQueue::with_capacity(1024);

    queue.enqueue(10);
    assert_eq!(queue.dequeue(), 10);
}

proves almost nothing about concurrency.

We need multiple producers and consumers.

For example:

use std::sync::Arc;
use std::thread;

let queue = Arc::new(
    WaitFreeQueue::with_capacity(1024)
);

let mut handles = Vec::new();

for producer_id in 0..4 {
    let queue = Arc::clone(&queue);

    handles.push(thread::spawn(move || {
        for i in 0..10_000 {
            queue.enqueue(
                producer_id * 10_000 + i
            );
        }
    }));
}

Then consumers drain the queue.

We can verify:

number produced == number consumed

and:

no values duplicated

and:

no values lost

Stress Testing

Concurrency bugs are often probabilistic.

The code may pass:

10 runs

and fail:

run 11,482

You want tests that create different scheduling conditions.

For example:

Producer A
    |
    | sleep
    v
Producer B
    |
    v
Consumer A
    |
    v
Producer A resumes

The scheduler becomes part of the adversary.

A good concurrent test attempts to make timing unpredictable.

Loom and Model Checking

For serious Rust concurrency work, tools such as loom are extremely useful.

Instead of merely executing one scheduling order:

A -> B -> C

a model checker can explore many possible interleavings:

A -> B -> C
A -> C -> B
B -> A -> C
B -> C -> A
C -> A -> B
C -> B -> A

The number of schedules grows quickly.

That's precisely why concurrent algorithms are difficult to reason about manually.

The Bigger Lesson

Building this queue teaches something more important than queues.

It teaches that concurrency is fundamentally about ownership transitions.

At every moment, someone needs to answer:

Who owns this memory?

For an empty slot:

producer owns it

For a published slot:

consumer owns it

For a consumed slot:

producer owns it again

The queue is essentially a machine for transferring ownership between threads.

Rust's ownership system operates at the language level.

Our queue's sequence numbers implement a second ownership system at runtime.

That's the beautiful part.

Rust Is Not Making the Algorithm Easy

Rust prevents many ordinary memory bugs.

It does not prove that your lock-free algorithm is correct.

This code:

unsafe {
    (*slot.value.get()).write(value);
}

is not automatically safe because it compiles.

We need an invariant.

Something like:

A producer may write to a slot only after observing the sequence corresponding to its exclusive logical position.

And:

A consumer may read a slot only after observing the producer's release publication.

And:

A slot cannot be simultaneously owned by a producer and consumer.

Those statements form the beginning of the correctness proof.

A Useful Mental Model

When designing concurrent data structures, think in terms of:

STATE
  |
  v
WHO OWNS IT?
  |
  v
WHAT ATOMIC EVENT TRANSFERS OWNERSHIP?
  |
  v
WHAT MEMORY ORDERING MAKES THE DATA VISIBLE?
  |
  v
CAN ANOTHER THREAD GET STUCK?
  |
  v
IS THE NUMBER OF STEPS BOUNDED?

That final question separates wait-free algorithms from many merely lock-free designs.

Wait-Free vs Lock-Free

Let's make the distinction painfully clear.

Mutex queue

Thread A
   |
 locks
   |
   X
   |
 Thread B waits

Progress depends on lock ownership.

Lock-free queue

Thread A
   |
   X
   |
Thread B -----> succeeds

The system makes progress.

Wait-free queue

Thread A
   |
   X

Thread B -----> succeeds

Thread C ----------> succeeds

Thread D ----------------> succeeds

Every operation has a bounded progress guarantee.

The difference is not cosmetic.

It is mathematical.

When Should You Use Wait-Free Structures?

Almost never by default.

If you are writing an ordinary web application:

HTTP request
     |
     v
database
     |
     v
response

you probably do not need a wait-free queue.

A mutex-backed queue may be simpler and perfectly adequate.

Wait-free algorithms become interesting when latency and contention matter enough to justify the complexity.

Examples include:

real-time systems
high-frequency messaging
network runtimes
telemetry
game engines
audio systems
embedded systems
low-latency trading infrastructure
operating-system components

The key is not:

“Wait-free is faster.”

It isn't necessarily.

The key is:

“Wait-free gives a stronger bound on progress.”

That's a different property.

The Hidden Cost of Wait-Free

Imagine two systems.

System A:

average latency: 2 μs
worst case: 500 μs

System B:

average latency: 3 μs
worst case: 5 μs

If you care about real-time behavior, System B may be vastly more valuable.

This is why worst-case guarantees matter.

Average performance tells you what normally happens.

Wait-free reasoning asks:

What can happen in the worst case?

That is a much more demanding question.

What We Built

Our conceptual queue contains:

             WaitFreeQueue
                  |
       +----------+----------+
       |                     |
 enqueue_pos            dequeue_pos
       |                     |
       v                     v
   producers             consumers
       |                     |
       +----------+----------+
                  |
                  v
            ring buffer
                  |
          +-------+-------+
          |       |       |
          v       v       v
        Slot    Slot    Slot
          |       |       |
          +-------+-------+
                  |
                  v
           sequence numbers

Each slot contains:

sequence
value

The sequence identifies the logical generation of the slot.

The atomic positions allocate logical queue positions.

Release/acquire ordering transfers visibility.

And bounded capacity avoids allocation during ordinary operations.

The Most Important Caveat

If you take one thing from this article, make it this:

An atomic queue is not automatically a wait-free queue.

A queue can be:

mutex-based
lock-free
wait-free

and all three can be implemented correctly.

The distinction is the progress guarantee.

A loop such as:

while !try_operation() {
    spin_loop();
}

should immediately make you ask:

Is the number of iterations bounded?

If the answer is no, you should be extremely cautious about calling the algorithm wait-free.

This is one of the easiest mistakes to make when discussing concurrent data structures.

Where to Go Next

Once you understand this queue, there are several deeper directions.

You can explore:

Michael-Scott queues

for dynamically allocated lock-free queues.

Then:

hazard pointers

for safe memory reclamation.

Then:

epoch-based reclamation

for another approach to managing retired nodes.

Then:

helping

for constructing genuinely wait-free algorithms.

And eventually:

linearizability

which gives you a formal way to reason about whether concurrent operations appear to occur atomically.

That's where concurrent programming starts looking less like ordinary application development and more like mathematics.

Final Thought

The first time you build a concurrent queue, you think the difficult part is storing the data.

It isn't.

The difficult part is convincing multiple CPUs that they agree about reality.

One CPU says:

"This slot is mine."

Another says:

"I am reading it."

A third may be delayed for milliseconds.

The cache coherence protocol is doing things underneath you.

The compiler is allowed to reorder operations unless synchronization prevents it.

The operating system may pause a thread at exactly the wrong moment.

And yet your data structure must remain correct.

That's what makes wait-free programming fascinating.

You aren't merely writing code.

You're defining a protocol between independent machines that happen to share memory.

Rust gives you powerful tools for expressing that protocol:

AtomicUsize
UnsafeCell
MaybeUninit
Acquire
Release

But the real skill is learning what those tools mean together.

A wait-free queue is therefore more than a queue.

It is a lesson in distributed systems disguised as shared memory.

There are multiple participants.

They cannot assume another participant will respond.

They communicate through carefully defined state transitions.

They need explicit ownership.

They need ordering guarantees.

And the strongest algorithms make progress even when one participant disappears.

That sounds a lot like distributed computing.

Except the network is your CPU's memory subsystem.

And the latency is measured in nanoseconds.

That is the strange beauty of concurrent programming.

The moment you stop asking “does this code work?” and start asking “what is the worst thing another thread can do to me?” — you are beginning to think like a systems programmer.

Building a Constraint Solver: Teaching a Computer How to Reason About Impossible Things

Derek Mwale — Thu, 27 Aug 2026 09:40:04 +0000

There is a particular kind of programming problem that feels almost philosophical.

You don't know the answer.

You don't even know exactly what the answer should look like.

You only know the rules.

A teacher needs to schedule classes without putting two classes in the same room.

A Sudoku board needs every row, column, and region to contain unique numbers.

A delivery system needs to assign vehicles without exceeding capacity.

A university needs to assign students to courses while respecting prerequisites and room limits.

A compiler needs to satisfy constraints while generating code.

A cloud scheduler needs to place workloads onto machines without exceeding CPU and memory.

A configuration system needs to find a combination of versions that are mutually compatible.

In all of these problems, you aren't really asking:

"What is the answer?"

You're asking:

"Can I find an assignment of values that satisfies every rule?"

That is the fundamental idea behind a constraint solver.

And once you understand how to build one, you begin to see something fascinating.

A constraint solver is a tiny machine for reasoning.

You give it variables.

You give it possible values.

You give it rules.

And the machine searches the space of possibilities until it discovers an assignment that doesn't violate those rules.

The architecture looks deceptively simple:

                 CONSTRAINT SOLVER

                    Variables
                        |
                        v
                 Possible Values
                        |
                        v
                   Constraints
                        |
                        v
                 +-------------+
                 |   Search    |
                 +-------------+
                        |
              +---------+---------+
              |                   |
           Violation            Success
              |                   |
              v                   v
          Backtrack             Solution
              |
              v
        Try another value

But inside that simple diagram is one of the most beautiful ideas in computer science:

Instead of explicitly knowing the answer, we define the space of valid answers and systematically eliminate everything impossible.

Let's build one.

1. What Exactly Is a Constraint Solver?

A constraint satisfaction problem, commonly called a CSP, consists of three fundamental components:

CSP = Variables + Domains + Constraints

Variables

Variables are the things we need to determine.

For example:

X
Y
Z

Domains

A domain is the set of values a variable can take.

For example:

X ∈ {1, 2, 3}
Y ∈ {1, 2, 3}
Z ∈ {1, 2, 3}

Constraints

Constraints define which combinations are allowed.

For example:

X != Y
Y != Z
X != Z

The solver's job is to find:

X = ?
Y = ?
Z = ?

such that every constraint is satisfied.

One valid solution is:

X = 1
Y = 2
Z = 3

The important thing is that we never explicitly told the computer:

X = 1
Y = 2
Z = 3

We gave it the rules.

The solver discovered the assignment.

That's the entire game.

2. A Constraint Solver Is Basically Controlled Brute Force

Let's start with the simplest possible approach.

Suppose:

X ∈ {1, 2, 3}
Y ∈ {1, 2, 3}

and:

X != Y

There are nine possible assignments:

(1,1)
(1,2)
(1,3)
(2,1)
(2,2)
(2,3)
(3,1)
(3,2)
(3,3)

The solver can simply try them.

             Search Space

             X = 1
            /     \
         Y=1      Y=2
         ❌        ✓

             X = 2
            /     \
         Y=1      Y=2
         ✓         ❌

             X = 3
            /     \
         Y=1      Y=2
         ✓         ✓

This is brute force.

And brute force is actually an excellent place to start.

Why?

Because a constraint solver isn't magic.

It is fundamentally a search algorithm.

The intelligence comes from learning how to avoid searching parts of the space that cannot possibly produce a solution.

That is where things become interesting.

3. The Search Tree

Imagine three variables:

A
B
C

Each has:

{1, 2, 3}

The complete search space contains:

3 × 3 × 3 = 27

possibilities.

The search tree looks like:

                         Start
                           |
          +----------------+----------------+
          |                |                |
        A = 1            A = 2            A = 3
       / | \             / | \             / | \
     B=1 B=2 B=3       B=1 B=2 B=3       B=1 B=2 B=3
      |    |    |        |    |    |        |    |    |
      C    C    C        C    C    C        C    C    C

Even with only three variables, the tree grows quickly.

If there are:

10 variables
10 values each

then the naive search space is:

10^10

That's:

10,000,000,000

possibilities.

And that's a tiny CSP.

This is why constraint solving is fundamentally an exercise in search-space reduction.

4. Backtracking

The first serious algorithm we need is backtracking.

The idea is simple.

Assign a value.

Check whether the partial assignment is still valid.

If it is, continue.

If it isn't, undo the assignment and try another value.

Choose value
     |
     v
Does it violate a constraint?
     |
   +---+---+
   |       |
  Yes      No
   |       |
Backtrack  Continue

This is much smarter than blindly generating complete assignments.

Consider:

A != B
B != C
A != C

We assign:

A = 1

Then:

B = 1

Immediately:

A == B

which violates:

A != B

There is no reason to assign C.

The entire subtree beneath:

A = 1
B = 1

can be discarded.

This is called pruning.

And pruning is where constraint solvers become powerful.

5. Building the First Solver in Python

Let's make this concrete.

We'll build a small generic solver.

First, variables:

variables = ["A", "B", "C"]

Domains:

domains = {
    "A": [1, 2, 3],
    "B": [1, 2, 3],
    "C": [1, 2, 3],
}

Now our constraint:

def all_different(assignment):
    values = list(assignment.values())
    return len(values) == len(set(values))

And our solver:

def solve(variables, domains, assignment=None):
    if assignment is None:
        assignment = {}

    if len(assignment) == len(variables):
        return assignment.copy()

    variable = next(
        v for v in variables
        if v not in assignment
    )

    for value in domains[variable]:
        assignment[variable] = value

        if all_different(assignment):
            result = solve(
                variables,
                domains,
                assignment
            )

            if result is not None:
                return result

        del assignment[variable]

    return None

We can run:

solution = solve(
    variables,
    domains
)

print(solution)

And get something like:

{'A': 1, 'B': 2, 'C': 3}

We have built a constraint solver.

It's not sophisticated.

But conceptually, this is the foundation of many much more advanced systems.

6. The Recursive Mental Model

The most important part of that implementation isn't the Python syntax.

It's this:

for value in domains[variable]:
    assignment[variable] = value

    if constraints_satisfied():
        result = solve(...)

        if result:
            return result

    del assignment[variable]

This is the heart of backtracking.

Think of recursion as moving deeper into the search tree.

solve()
   |
   +--- A=1
   |      |
   |      +--- B=1 ❌
   |      |
   |      +--- B=2
   |             |
   |             +--- C=1 ❌
   |             +--- C=2 ❌
   |             +--- C=3 ✓
   |
   +--- A=2
   |
   +--- A=3

When we hit a violation, recursion returns.

That is the "backtrack."

We go back up the tree and try something else.

7. Why Naive Backtracking Isn't Enough

Our solver works.

But it can still become ridiculously slow.

Imagine:

100 variables
10 values each

The theoretical search space is:

10^100

No computer is casually iterating through that.

The entire field of constraint solving is essentially about avoiding that explosion.

We need to make the search smarter.

The first major improvement is:

Choose the variable intelligently.

8. Minimum Remaining Values

Suppose we have:

A = {1, 2, 3, 4, 5}
B = {2}
C = {1, 2, 3}
D = {1, 2, 3, 4}

Which variable should we assign first?

A naive solver might choose:

But that's a bad choice.

B has only one possible value.

So choose:

This strategy is called Minimum Remaining Values, or MRV.

The idea:

Choose the variable with the smallest remaining domain.

A → 5 possibilities
B → 1 possibility
C → 3 possibilities
D → 4 possibilities

Choose B.

Why?

Because variables with tiny domains are the most constrained.

If B cannot be assigned, we'd rather discover that immediately.

This is an example of the fail-first principle:

Try to discover failure as early as possible.

9. Implementing MRV

We can replace:

variable = next(
    v for v in variables
    if v not in assignment
)

with:

unassigned = [
    v for v in variables
    if v not in assignment
]

variable = min(
    unassigned,
    key=lambda v: len(domains[v])
)

Now our solver makes a more intelligent decision.

But we can go further.

10. Forward Checking

Suppose:

A != B

and:

A ∈ {1, 2, 3}
B ∈ {1, 2, 3}

We assign:

A = 1

We already know:

B != 1

So why leave:

B = {1, 2, 3}

We can immediately reduce it:

B = {2, 3}

This is called forward checking.

The assignment pushes information forward through the problem.

A = 1
 |
 +----> B cannot be 1
 |
 +----> C cannot be 1
 |
 +----> D cannot be 1

Now the future search has fewer possibilities.

This is one of the fundamental patterns of constraint programming:

Every decision should eliminate impossible futures.

11. Constraint Propagation

Forward checking is one form of constraint propagation.

Imagine:

A < B
B < C

with:

A = {1, 2, 3}
B = {1, 2, 3}
C = {1, 2, 3}

The constraints imply relationships between the domains.

For example, if:

A = 3

then:

B > 3

which is impossible.

Therefore:

A = 3

can be eliminated before we ever assign it.

Propagation tries to derive these consequences automatically.

Conceptually:

Initial domains
      |
      v
Apply constraints
      |
      v
Reduce domains
      |
      v
Apply constraints again
      |
      v
Reduce domains
      |
      v
Fixed point

This process can dramatically shrink the search space.

12. Arc Consistency

One famous idea in CSPs is arc consistency.

Consider:

X < Y

with:

X = {1, 2, 3}
Y = {1, 2}

Can:

X = 3

possibly work?

No.

There is no value in Y greater than 3.

So remove:

from X.

Now:

X = {1, 2}
Y = {1, 2}

This seems small.

But imagine hundreds of variables with interconnected constraints.

One domain reduction can trigger another.

X shrinks
   |
   v
Y shrinks
   |
   v
Z shrinks
   |
   v
W becomes impossible

A solver can propagate these consequences before committing to a deep search.

13. The Constraint Graph

A useful way to visualize a CSP is as a graph.

Suppose:

A != B
B != C
C != D
A != D

We can represent it as:

       A -------- B
       |           |
       |           |
       D -------- C

Each node is a variable.

Each edge represents a relationship.

This graph gives us another powerful heuristic.

When choosing the next variable, we can consider not only how constrained its domain is, but also how many other variables depend on it.

That's the degree heuristic.

A highly connected variable can influence a large portion of the search.

So we may prefer it.

14. Variable Ordering Is a Superpower

Consider two strategies.

Strategy A

A
B
C
D
E
F

Strategy B

Most constrained
        ↓
Most connected
        ↓
Most likely to fail

Both are correct.

But their performance can be radically different.

This is an important lesson in algorithms:

The order in which you explore a search space can matter more than the search algorithm itself.

Two solvers can use the same backtracking algorithm and have dramatically different runtimes simply because they make different decisions about what to explore first.

15. Value Ordering

Choosing the variable is only half the problem.

We also need to decide which value to try first.

Suppose:

X = {1, 2, 3}

Which should we try?

One strategy is the least-constraining value.

Choose the value that eliminates the fewest possibilities from neighboring variables.

Imagine:

X = 1

eliminates 5 possibilities.

While:

X = 2

eliminates only 1.

Try:

X = 2

first.

The intuition is:

Keep the future as flexible as possible.

Now we have two complementary ideas:

MRV:
Choose the most constrained variable.

LCV:
Choose the least constraining value.

Together they can make backtracking dramatically smarter.

16. A Better Solver Architecture

Our solver is starting to become a real engine.

Conceptually:

                  Solver
                     |
          +----------+----------+
          |                     |
       Choose Variable      Choose Value
          |                     |
         MRV                    LCV
          |                     |
          +----------+----------+
                     |
                     v
                 Assignment
                     |
                     v
              Propagate Rules
                     |
              +------+------+
              |             |
            Valid         Conflict
              |             |
              v             v
           Continue      Backtrack

This is the architecture behind the basic idea.

17. Let's Build a More Generic Constraint API

Hardcoding all_different() isn't enough.

We want a reusable solver.

Let's create a constraint interface:

class Constraint:
    def __init__(self, variables):
        self.variables = variables

    def satisfied(self, assignment):
        raise NotImplementedError

Then:

class NotEqualConstraint(Constraint):
    def __init__(self, a, b):
        super().__init__([a, b])
        self.a = a
        self.b = b

    def satisfied(self, assignment):
        if self.a not in assignment:
            return True

        if self.b not in assignment:
            return True

        return assignment[self.a] != assignment[self.b]

Now we can define:

constraints = [
    NotEqualConstraint("A", "B"),
    NotEqualConstraint("B", "C"),
]

The solver doesn't need to know what the constraints mean.

That's important.

The solver handles:

search
assignment
backtracking

while constraints handle:

domain-specific logic

This separation makes the system extensible.

18. The Solver Core

We can now write something like:

def consistent(variable, assignment, constraints):
    for constraint in constraints:
        if variable in constraint.variables:
            if not constraint.satisfied(assignment):
                return False

    return True

And:

def solve(
    variables,
    domains,
    constraints,
    assignment=None
):
    if assignment is None:
        assignment = {}

    if len(assignment) == len(variables):
        return assignment.copy()

    unassigned = [
        v for v in variables
        if v not in assignment
    ]

    variable = min(
        unassigned,
        key=lambda v: len(domains[v])
    )

    for value in domains[variable]:
        assignment[variable] = value

        if consistent(
            variable,
            assignment,
            constraints
        ):
            result = solve(
                variables,
                domains,
                constraints,
                assignment
            )

            if result is not None:
                return result

        del assignment[variable]

    return None

This is still a small solver.

But notice what happened.

We separated:

Problem definition

from:

Search engine

That is the beginning of a real constraint programming system.

19. Sudoku Is a Constraint Problem

Now let's apply the idea to something more interesting.

Sudoku.

A Sudoku board has:

81 variables

Each cell is a variable.

Each cell has a domain:

{1,2,3,4,5,6,7,8,9}

Then we have constraints.

Every row:

All different

Every column:

All different

Every 3×3 region:

All different

The problem becomes:

81 variables
+
81 domains
+
row constraints
+
column constraints
+
region constraints

The solver doesn't need to know that this is "Sudoku."

It only needs to know the constraints.

That's a beautiful abstraction.

20. Sudoku as a Graph

Imagine every cell as a node.

Two cells are connected if they cannot contain the same number.

       Cell A -------- Cell B
         |               |
         |               |
         |               |
       Cell C -------- Cell D

Each cell has roughly 20 peers in a standard Sudoku board.

When:

A = 5

the solver can remove:

from every neighboring cell.

If one neighboring cell becomes:

{5}

we've discovered another assignment.

That assignment propagates.

Sudoku is therefore an excellent demonstration of constraint propagation.

21. The Search Explosion

Why can't we just brute-force Sudoku?

Because each empty cell could initially contain nine values.

For 81 cells:

9^81

possibilities.

That's an absurdly large number.

But Sudoku constraints eliminate enormous portions of that space.

The solver isn't really exploring:

9^81

because propagation continually reduces the possibilities.

This is the essence of constraint solving:

Huge theoretical space
        |
        v
Constraint propagation
        |
        v
Smaller search space
        |
        v
Heuristics
        |
        v
Even smaller search
        |
        v
Backtracking
        |
        v
Solution

22. Hard Constraints vs Soft Constraints

So far we've assumed that every constraint must be satisfied.

These are hard constraints.

For example:

room_capacity >= students

must always be true.

But real-world scheduling often contains preferences.

For example:

Professor prefers morning classes.

That's not necessarily mandatory.

This introduces soft constraints.

Now we're no longer simply asking:

"Is there a solution?"

We're asking:

"What is the best solution?"

This changes the problem.

We can define a score:

score(solution)

and search for:

max(score)

or:

min(cost)

Now the solver starts looking like an optimization engine.

23. Constraint Solving and Optimization

Suppose we're scheduling meetings.

Hard constraints:

No overlapping meetings.
Room capacity must be sufficient.
Employees cannot be in two places simultaneously.

Soft constraints:

Prefer morning.
Prefer certain rooms.
Minimize walking distance.
Prefer employee availability.

Now we can represent the objective:

Cost =
    scheduling_penalty
  + room_penalty
  + travel_penalty
  + availability_penalty

The solver attempts to minimize:

Cost

Suddenly our little backtracking algorithm has become a primitive optimization engine.

This is one reason constraint programming is so useful.

24. Constraint Solvers Are Everywhere

Once you understand the pattern, you'll start seeing CSPs everywhere.

Scheduling

People
Rooms
Time slots
Resources

Routing

Vehicles
Locations
Capacity
Time windows

Configuration management

Package A requires B >= 2
Package C conflicts with D
Package E requires F

Cloud infrastructure

CPU
Memory
Availability zones
Affinity
Anti-affinity

Compilers

Registers
Instructions
Dependencies
Scheduling constraints

Network design

Capacity
Latency
Topology
Routing rules

Games

Legal moves
Board state
Resource limits
Victory conditions

Artificial intelligence

Variables
Possible states
Relationships
Objectives

A surprising amount of software can be reframed as:

Find an assignment satisfying these rules.

25. Constraint Propagation as Information Flow

One of the deepest ways to think about a solver is not as a brute-force machine, but as an information propagation system.

Imagine:

A = 5

and:

A != B

Then:

B != 5

Now suppose:

B ∈ {5, 6}

After propagation:

B = {6}

Therefore:

B = 6

Then perhaps:

B != C

which tells us:

C != 6

One fact creates another.

A = 5
   |
   v
B != 5
   |
   v
B = 6
   |
   v
C != 6
   |
   v
C = 7
   |
   v
...

This is almost like a chain reaction.

The solver keeps asking:

"What else becomes impossible because of what I just learned?"

That's constraint propagation.

26. Detecting Failure Early

Suppose a domain becomes empty:

X = {}

This means:

There is no possible value for X under the current partial assignment.

We don't need to search further.

We immediately backtrack.

X domain
{1,2,3}
   |
constraint
   v
{2,3}
   |
constraint
   v
{3}
   |
constraint
   v
{}
   |
   v
CONFLICT
   |
   v
BACKTRACK

An empty domain is one of the cleanest signals a solver can receive.

It means:

This branch of the search tree is mathematically impossible.

27. Conflict-Directed Thinking

As solvers become more sophisticated, they don't merely say:

"Something failed."

They try to understand:

"Why did it fail?"

Suppose:

A = 1
B = 1

and:

A != B

The conflict involves:

A
B

There is no reason to reconsider unrelated variable:

This idea leads toward more advanced techniques such as conflict-directed backtracking and conflict-driven search.

Instead of blindly undoing the most recent assignment, the solver can jump back to the decision responsible for the conflict.

This can dramatically improve performance on difficult problems.

28. Memoization

Sometimes different paths lead to the same state.

Imagine the solver reaches:

A = 1
B = 2

through one path.

Later, another path reaches the exact same relevant state.

If we've already proven:

A = 1, B = 2

cannot produce a solution, why solve it again?

We can cache failed states.

failed_states = set()

Then:

state = tuple(sorted(assignment.items()))

if state in failed_states:
    return None

After failure:

failed_states.add(state)

This is essentially memoization applied to search.

But there is a subtlety.

For large problems, storing entire states can consume enormous memory.

Again, engineering becomes a trade-off.

29. Bitsets Make Constraint Solvers Fast

If domains contain small integers, we can represent them using bits.

For Sudoku:

1 2 3 4 5 6 7 8 9

can be represented by a 9-bit integer.

For example:

000001111

could represent:

{1,2,3,4}

Removing a value becomes a bit operation.

Intersection becomes:

domain_a & domain_b

Union:

domain_a | domain_b

Difference:

domain_a & ~domain_b

Now domain operations become extremely cheap.

This is one reason highly optimized constraint solvers often look very different from their educational Python implementations.

The underlying mathematics stays the same.

The representation changes.

30. SAT: Constraint Solving Taken to Another Level

There is another major family of solvers:

SAT solvers.

SAT asks:

Is there an assignment of Boolean variables that makes a logical formula true?

For example:

A OR B

and:

NOT A OR C

A solution might be:

A = false
B = true
C = true

SAT solvers are extraordinarily powerful.

Modern SAT solving has developed sophisticated techniques around:

conflict analysis
clause learning
watched literals
decision heuristics
non-chronological backtracking
restarts

A general constraint solver and a SAT solver are different tools, but conceptually they share the same DNA:

Decide
  |
Propagate
  |
Conflict?
  |
Learn
  |
Backtrack
  |
Decide again

That pattern is incredibly powerful.

31. Constraint Programming vs Brute Force

It is tempting to describe a constraint solver as:

"A smarter brute-force algorithm."

That's not completely wrong.

But it's incomplete.

The deeper idea is:

Search plus inference.

Brute force says:

Try everything.

Constraint solving says:

Infer what cannot work.
Don't search it.

That distinction is enormous.

Suppose a search space contains:

1,000,000,000 possibilities

and propagation proves that:

999,999,000

are impossible without exploring them individually.

Now the problem isn't really a billion possibilities anymore.

The solver has compressed the search space using knowledge.

That's the magic.

32. A Production-Grade Architecture

If I were designing a serious constraint-solving library, I'd separate it into layers.

+------------------------------------------------+
|                 User API                       |
+------------------------------------------------+
|           Problem Definition                   |
+------------------------------------------------+
|       Variables / Domains / Constraints        |
+------------------------------------------------+
|          Constraint Propagation                |
+------------------------------------------------+
|            Search Heuristics                   |
+------------------------------------------------+
|         Backtracking Engine                    |
+------------------------------------------------+
|        Conflict / Learning Layer               |
+------------------------------------------------+
|          Domain Representation                 |
+------------------------------------------------+
|        Low-Level Data Structures               |
+------------------------------------------------+

The user should be able to say something like:

solver = Solver()

x = solver.variable(
    "x",
    domain=range(1, 10)
)

y = solver.variable(
    "y",
    domain=range(1, 10)
)

solver.add(x != y)

solution = solver.solve()

The user doesn't need to understand recursion.

They shouldn't need to.

The engine handles it.

33. Instrumentation Is Essential

A solver can return the correct answer and still be terrible.

You need to know what it did.

Track metrics such as:

nodes explored
backtracks
constraint checks
propagations
domain reductions
conflicts
maximum depth
solutions found
runtime

For example:

Solver Statistics

Nodes explored:      42,182
Backtracks:          8,391
Propagations:        190,442
Conflicts:           8,391
Solution time:       0.81s

Now you can compare algorithms.

Maybe MRV reduces:

nodes:
42,182 → 8,200

That's evidence.

Performance engineering should always have evidence.

34. Visualizing the Search

One of my favorite ways to understand a solver is to visualize its search tree.

                        START
                          |
                +---------+---------+
                |                   |
              X = 1               X = 2
                |                   |
          +-----+-----+             |
          |           |             |
        Y = 1       Y = 2         Y = 1
          |           |             |
          X           ✓             X
       CONFLICT                  CONFLICT

You can color nodes conceptually as:

✓ = solution
✗ = conflict
○ = unexplored

Then you can watch the solver prune branches.

A good solver is essentially trying to make this tree look less like:

████████████████████████████

and more like:

███
 █
 ██
  █
   ✓

It wants to reach a solution without exploring unnecessary territory.

35. The Strange Connection to AI

Constraint solving has an interesting relationship with artificial intelligence.

Machine learning often works by learning patterns from data.

Constraint solving works by reasoning from explicit rules.

Machine Learning:

Data
 |
 v
Learn Model
 |
 v
Prediction


Constraint Solver:

Rules
 |
 v
Search + Propagation
 |
 v
Valid Assignment

But modern AI systems increasingly combine both.

Imagine an optimization problem where a machine-learning model predicts:

"This variable is probably the best one to assign next."

The ML model provides a heuristic.

The constraint solver guarantees correctness.

That combination is powerful:

Machine Learning
       |
       v
Search Heuristic
       |
       v
Constraint Solver
       |
       v
Guaranteed-valid solution

This is one direction where classical symbolic reasoning and modern AI can complement each other.

36. The Most Important Optimization: Don't Search What You Can Prove

This is the philosophy I'd keep in mind when building a solver.

Suppose you have:

1,000 possible values.

If you can prove:

700 are impossible

don't search them.

If you can prove another:

200 are impossible

don't search those either.

Now you only have:

possibilities.

Constraint propagation is therefore a form of computational compression.

You're compressing the search space by reasoning about it.

That is why the best constraint solvers don't feel like brute force.

They feel like deduction.

37. Building the Solver Incrementally

If I were actually implementing this project, I wouldn't start with SAT solving or advanced propagation.

I'd build it in stages.

Version 1

Variables
Domains
Constraints
Backtracking

Version 2

MRV

Version 3

Forward checking

Version 4

Constraint propagation

Version 5

Degree heuristic
LCV

Version 6

Conflict detection

Version 7

Memoization

Version 8

Bitset domains

Version 9

Conflict-directed backtracking

Version 10

Optimization objectives

This progression is valuable because every version teaches you something.

You aren't just building software.

You're discovering why the sophisticated algorithms exist.

38. Testing a Constraint Solver

Testing a solver is different from testing ordinary business logic.

You need to test:

Satisfiable problems

A != B
A = {1,2}
B = {1,2}

Expected:

Solution exists.

Unsatisfiable problems

A = 1
A = 2

Expected:

No solution.

Multiple solutions

Verify that returned assignments satisfy every constraint.

Empty domains

The solver should fail gracefully.

Large domains

Test performance.

Deep recursion

Ensure the implementation doesn't collapse under realistic problem sizes.

And most importantly:

for constraint in constraints:
    assert constraint.satisfied(solution)

A solver's answer is worthless if it violates the rules.

Correctness comes before speed.

39. The Beautiful Thing About Constraints

There is something almost philosophical about constraint programming.

You don't tell the machine:

"Do this."

You tell it:

"This is allowed."

And:

"This is forbidden."

Then you let it reason.

That changes the programming model.

Traditional imperative programming often looks like:

Step 1
Step 2
Step 3
Step 4
Step 5

Constraint programming looks more like:

Here are the variables.

Here are their possible values.

Here are the rules.

Find something that satisfies them.

The computer determines the path.

You're describing the destination indirectly.

That's a powerful abstraction.

40. Final Architecture

At the end of our journey, the solver looks something like this:

                         PROBLEM
                            |
                            v
              +--------------------------+
              | Variables + Domains      |
              +--------------------------+
                            |
                            v
              +--------------------------+
              |      Constraints         |
              +--------------------------+
                            |
                            v
              +--------------------------+
              | Constraint Propagation   |
              +--------------------------+
                            |
                            v
              +--------------------------+
              | Variable Selection       |
              | MRV / Degree             |
              +--------------------------+
                            |
                            v
              +--------------------------+
              | Value Selection           |
              | LCV / Heuristics         |
              +--------------------------+
                            |
                            v
                     Assignment
                            |
                            v
                     Propagation
                            |
                 +----------+----------+
                 |                     |
              Conflict              Valid
                 |                     |
                 v                     v
             Backtrack             Continue
                 |                     |
                 +----------<----------+
                            |
                            v
                       Solution

This architecture is deceptively small.

But it contains the core ideas behind some very sophisticated optimization and reasoning systems.

Conclusion: Build Machines That Know What Not to Try

The first constraint solver I would build is intentionally stupid.

It should brute-force.

It should make bad decisions.

It should backtrack constantly.

Because then you can watch it suffer.

And that suffering teaches you something.

You add MRV.

The search shrinks.

You add forward checking.

It shrinks again.

You add propagation.

It gets faster.

You add better value ordering.

Faster again.

You add conflict learning.

The solver starts jumping over entire regions of the search space.

Eventually, you realize something profound:

The fastest search isn't necessarily the one that searches faster.

It's the one that searches less.

That is the central idea behind constraint solving.

A naive program asks:

"What should I try next?"

A constraint solver asks:

"What can I prove I don't need to try?"

That difference is enormous.

And it appears everywhere.

In scheduling.

In databases.

In compilers.

In logistics.

In cloud infrastructure.

In configuration management.

In games.

In optimization.

In artificial intelligence.

In mathematics.

The world is full of systems where the answer isn't explicitly known, but the rules are.

And whenever you have:

Variables
+
Possible values
+
Rules

you may have the beginning of a constraint satisfaction problem.

The computer doesn't need you to hand it the answer.

You can give it something more interesting.

Give it the rules.

Then teach it how to reason.

And suddenly, a collection of variables becomes a search space.

A collection of rules becomes a mathematical structure.

And a recursive function becomes something much more interesting:

a machine that systematically explores possibility while learning what is impossible.

That's a constraint solver.

And once you understand how to build one, you start seeing programming differently.

Because a surprising amount of software engineering isn't really about telling computers what to do.

It's about defining the boundaries of what they are allowed to do—and giving them enough intelligence to find the path through the space that remains.

Making Django 10x Faster: The Performance Engineering Playbook

Derek Mwale — Thu, 27 Aug 2026 09:30:18 +0000

There is a point in almost every Django project where the application starts behaving differently.

At 100 users, everything feels instant.

At 1,000 users, you notice a few slow pages.

At 10,000 users, someone opens a monitoring dashboard and says:

“Why is this endpoint taking 1.8 seconds?”

Then somebody adds Redis.

Then somebody adds more Gunicorn workers.

Then somebody increases the database server.

Then somebody adds Nginx caching.

And somehow, three weeks later, the application is still slow.

This is one of the most interesting things about performance engineering:

A slow application is rarely caused by one slow thing.

It is usually caused by a chain of small decisions.

A query that takes 40 ms.

A serializer that performs another query.

A template that triggers lazy evaluation.

A loop that performs 500 database calls.

A missing database index.

A giant JSON response.

A synchronous external API call.

A badly configured connection pool.

A cache that is never actually hit.

Each problem looks harmless by itself.

Together, they create a system that feels broken.

The good news is that Django is already an extremely capable framework. You usually don't need to abandon Django, rewrite everything in Rust, or replace your database.

You need to understand what your application is actually doing.

And once you do that, making Django dramatically faster becomes less about magic and more about engineering.

This is how I think about it:

                 Django Performance

                       Request
                          |
                          v
                 +----------------+
                 |     Nginx      |
                 +----------------+
                          |
                          v
                 +----------------+
                 |    Gunicorn    |
                 +----------------+
                          |
                          v
                 +----------------+
                 |     Django     |
                 |                |
                 | Middleware     |
                 | Views          |
                 | Serializers    |
                 | Templates      |
                 +----------------+
                    /          \
                   /            \
                  v              v
          +-------------+   +----------+
          | PostgreSQL  |   |  Redis   |
          +-------------+   +----------+
                 |
                 v
          External Services

The objective isn't simply to make Python execute faster.

The objective is to reduce the amount of work required to complete a request.

That's a completely different mindset.

1. First: Stop Guessing

The biggest performance mistake developers make is optimizing code before measuring it.

Someone says:

“Django is slow.”

That's not a diagnosis.

Which part is slow?

The database?

Python?

Serialization?

Network latency?

Template rendering?

External APIs?

Lock contention?

Connection establishment?

Garbage collection?

CPU saturation?

Memory pressure?

Disk I/O?

You cannot optimize a mystery.

Start with measurements.

A useful mental model is:

Request latency
     =
network
+
middleware
+
application code
+
database
+
external services
+
serialization
+
response generation

If your endpoint takes 1,000 ms and PostgreSQL consumes 850 ms, optimizing Python by 50% changes almost nothing.

You reduced 150 ms to 75 ms.

Total:

850 + 75 = 925 ms

Congratulations.

You spent six hours making the application 7.5% faster.

But if you optimize the database from 850 ms to 100 ms:

100 + 150 = 250 ms

Now you've achieved something meaningful.

This is why profiling comes first.

2. Understand the Django Request

Before optimizing Django, understand what happens during a request.

A simplified request lifecycle looks like this:

HTTP Request
     |
     v
Web Server
     |
     v
Middleware
     |
     v
URL Resolver
     |
     v
View
     |
     +------> ORM ------> Database
     |
     +------> Cache
     |
     +------> External APIs
     |
     v
Serializer / Template
     |
     v
Middleware
     |
     v
HTTP Response

Every arrow represents potential latency.

Suppose your endpoint does this:

def dashboard(request):
    users = User.objects.all()
    orders = Order.objects.all()
    stats = requests.get("https://analytics.example.com/stats")

    return JsonResponse({
        "users": list(users),
        "orders": list(orders),
        "stats": stats.json(),
    })

It looks innocent.

But you've potentially created:

A database query for users.
A database query for orders.
A network request to another server.
Serialization work.
JSON encoding.
Memory allocation.
Network transfer.

And because the external request is synchronous, Django waits.

The browser waits.

The user waits.

Everything waits.

Performance engineering begins by asking:

What work can I remove?

That question is more powerful than:

How can I execute this work faster?

3. The Biggest Django Performance Killer: N+1 Queries

If there is one Django optimization that developers should memorize, it is this:

Watch your query count.

Consider:

orders = Order.objects.all()

for order in orders:
    print(order.customer.name)

If there are 1,000 orders, you might accidentally generate:

1 query
+
1,000 customer queries
=
1,001 queries

This is the infamous N+1 problem.

The solution is often:

orders = Order.objects.select_related("customer")

Now Django can fetch the related object efficiently.

Conceptually:

Bad:

Order 1 ---> Customer query
Order 2 ---> Customer query
Order 3 ---> Customer query
Order 4 ---> Customer query
...
Order 1000 -> Customer query


Good:

             +----------------------+
             |      One Query       |
             | orders + customers   |
             +----------------------+
                       |
                       v
                 Application

For foreign keys and one-to-one relationships, select_related() is often appropriate.

For many-to-many and reverse relationships, use prefetch_related().

Example:

orders = (
    Order.objects
    .select_related("customer")
    .prefetch_related("items")
)

Instead of:

Orders query
Customer query
Items query
Items query
Items query
...

you can reduce the database work dramatically.

This isn't a micro-optimization.

On real applications, eliminating N+1 queries can turn a painfully slow endpoint into a fast one.

4. Don't Fetch Columns You Don't Need

Developers frequently write:

users = User.objects.all()

when they only need:

id
username
email

If your model contains:

id
username
email
bio
avatar
address
preferences
metadata
created_at
updated_at
...

you're potentially moving unnecessary data from PostgreSQL into Python.

Use:

users = User.objects.only(
    "id",
    "username",
    "email",
)

Or use:

users = User.objects.values(
    "id",
    "username",
    "email",
)

The second approach is particularly useful when you don't need full model instances.

For example:

users = list(
    User.objects
    .filter(is_active=True)
    .values("id", "username")
)

You are telling the database:

Give me exactly what I need.

That principle is incredibly important.

5. Stop Using `len(queryset)` When You Mean `count()`

Suppose you want to know how many users exist.

Don't do:

users = User.objects.filter(is_active=True)

if len(users) > 100:
    ...

That can force Django to load the queryset.

Use:

if users.count() > 100:
    ...

Even better, if you only care whether something exists:

if users.exists():
    ...

Because:

if len(queryset):

and:

if queryset.exists():

are conceptually different operations.

The second asks the database a much cheaper question:

Does at least one row exist?

You don't need to download an entire dataset just to answer yes or no.

6. Database Indexes Are Performance Superpowers

Imagine a table containing 20 million users.

You execute:

User.objects.filter(email="derek@example.com").first()

If email isn't indexed, PostgreSQL may need to inspect a huge portion of the table.

With an index:

                 Query

                   |
                   v

          +----------------+
          | Email Index    |
          +----------------+
                   |
                   v
             Matching row

instead of:

Query
  |
  v
Row 1
Row 2
Row 3
Row 4
...
Row 20,000,000

You can define indexes in Django:

class User(models.Model):
    email = models.EmailField(unique=True)

    class Meta:
        indexes = [
            models.Index(fields=["email"]),
        ]

If you frequently filter by:

status

consider an index.

If you frequently query:

user_id, created_at

a composite index may be useful:

models.Index(
    fields=["user", "created_at"]
)

But don't blindly index everything.

Indexes have costs.

They consume storage.

They slow writes.

They increase maintenance overhead.

The correct question isn't:

“Should I index this column?”

It's:

“What query patterns does my application actually perform?”

7. Learn to Read SQL

Django's ORM is wonderful.

But abstraction can become dangerous when you stop understanding what happens underneath it.

This:

Order.objects.filter(
    customer__email=email,
    status="completed"
)

eventually becomes SQL.

You should know how to inspect that SQL.

For development:

print(
    Order.objects
    .filter(status="completed")
    .query
)

You can also use PostgreSQL's query analysis tools.

For example:

EXPLAIN ANALYZE
SELECT *
FROM orders
WHERE status = 'completed';

Now you're no longer guessing.

You're seeing what the database actually did.

You may discover:

Seq Scan

when you expected:

Index Scan

That's the moment performance engineering becomes real.

8. Optimize the Query, Not Just Django

Sometimes the ORM isn't the problem.

The query itself is.

Suppose you're calculating:

orders = Order.objects.filter(
    created_at__gte=start,
    created_at__lte=end,
)

and then Python performs:

total = 0

for order in orders:
    total += order.amount

Why transfer thousands of rows into Python?

Let PostgreSQL perform the aggregation.

from django.db.models import Sum

total = (
    Order.objects
    .filter(
        created_at__gte=start,
        created_at__lte=end,
    )
    .aggregate(total=Sum("amount"))
)

Now:

Before:

Database
   |
   | thousands of rows
   v
Python
   |
   v
Sum


After:

Database
   |
   | SUM(amount)
   v
One result

Databases are extremely good at data aggregation.

Use them.

9. Cache Expensive Work

Not everything needs to be calculated repeatedly.

Imagine your homepage calculates:

10,000 products
500 categories
Trending products
Popular searches
Recommended products
Statistics

and every visitor causes the same calculations.

That's wasteful.

A cache changes the architecture:

Request
   |
   v
Redis
   |
   +---- HIT ----> Return cached result
   |
   +---- MISS
          |
          v
       Django
          |
          v
       Database
          |
          v
       Store in Redis

With Django's cache framework:

from django.core.cache import cache

data = cache.get("homepage")

if data is None:
    data = build_homepage()
    cache.set("homepage", data, timeout=300)

Now thousands of requests may avoid the database entirely.

But caching isn't free.

You now have a new problem:

Cache invalidation.

Suppose you cache:

product:123

for 24 hours.

Then someone changes the product price.

Your users may see the old price.

So caching requires thinking about:

expiration
invalidation
consistency
cache keys
memory usage
stampedes

Caching is powerful because you're essentially saying:

I am willing to trade perfect freshness for lower computation.

For many read-heavy systems, that's a fantastic trade.

10. Cache at Multiple Layers

Caching doesn't have to exist only inside Django.

You can have:

Browser Cache
      |
      v
CDN
      |
      v
Reverse Proxy
      |
      v
Django Cache
      |
      v
Database

The further up the stack a request can be served, the less work your application performs.

Imagine a static image requested 100,000 times.

You don't want:

100,000 requests
        |
        v
Django

You want:

100,000 requests
        |
        v
CDN
        |
        v
Cached asset

Django shouldn't be processing things it doesn't need to process.

11. Pagination Is Not Optional

This is dangerous:

posts = Post.objects.all()

Then returning everything:

return JsonResponse({
    "posts": list(posts)
})

Today there are 500 posts.

Next year there are 500,000.

Your endpoint quietly becomes a disaster.

Use pagination.

Django provides pagination primitives:

from django.core.paginator import Paginator

posts = Post.objects.order_by("-created_at")

paginator = Paginator(posts, 50)

page = paginator.get_page(request.GET.get("page"))

Now the application only processes a manageable slice.

The architecture becomes:

500,000 records
       |
       v
Database
       |
       v
50 records
       |
       v
Django
       |
       v
Client

The client rarely needs half a million objects.

12. Don't Return Giant JSON Responses

A backend can be computationally fast and still feel slow.

Why?

Because network transfer matters.

Imagine your API returns:

{
  "users": [
    {
      "id": 1,
      "username": "...",
      "bio": "...",
      "metadata": "...",
      "history": "...",
      "preferences": "...",
      "orders": [...]
    }
  ]
}

If the frontend only needs:

{
  "id": 1,
  "username": "derek"
}

everything else is wasted bandwidth.

Good API design is performance engineering.

Return only what the client needs.

13. Be Careful With Serializers

Django REST Framework makes API development incredibly productive.

But serializers can hide expensive work.

For example:

class OrderSerializer(serializers.ModelSerializer):
    customer_name = serializers.CharField(
        source="customer.name"
    )

    class Meta:
        model = Order
        fields = [
            "id",
            "amount",
            "customer_name",
        ]

If you serialize 1,000 orders without:

.select_related("customer")

you can accidentally create an N+1 problem.

So your API code should think in pairs:

Serializer
    +
Queryset optimization

For example:

queryset = (
    Order.objects
    .select_related("customer")
    .prefetch_related("items")
)

Then:

serializer = OrderSerializer(
    queryset,
    many=True
)

The serializer should not be forced to repeatedly discover related objects through database calls.

14. Don't Block Django With External APIs

This is one of the easiest ways to make a fast backend feel slow.

Consider:

def dashboard(request):
    response = requests.get(
        "https://api.example.com/data"
    )

    return JsonResponse(response.json())

If the external service takes 2 seconds, your request takes roughly 2 seconds.

Your infrastructure might be perfect.

Your database might be perfect.

Your Python code might be perfect.

You're still waiting.

For independent work, consider asynchronous execution or background jobs.

Architecture:

HTTP Request
     |
     v
Django
     |
     +----> Queue ----> Worker ----> External API
     |
     v
Immediate Response

Tools such as Celery or other task queues can move slow work outside the request lifecycle.

For example:

send_report.delay(user_id)

instead of:

send_report(user_id)

The difference is architectural.

15. Async Django Is Useful—But Don't Worship Async

Django supports asynchronous views.

You can write:

async def dashboard(request):
    ...

This can be valuable when your workload is I/O-heavy.

For example:

Request
   |
   +----> API A
   |
   +----> API B
   |
   +----> API C

If those operations can be performed concurrently, asynchronous programming can reduce waiting.

Conceptually:

Synchronous:

API A: |------|
API B:        |------|
API C:              |------|

Total: ~3 seconds


Concurrent:

API A: |------|
API B: |------|
API C: |------|

Total: ~1 second

But async doesn't magically make CPU-bound Python faster.

If you're doing expensive computation:

for i in range(100_000_000):
    calculate(i)

making the function async doesn't suddenly make the CPU execute faster.

Async is primarily about efficiently managing waiting.

16. Gunicorn Workers Matter

Eventually you need to think about how Django processes requests.

A common deployment architecture looks like:

                Internet
                   |
                   v
                Nginx
                   |
          +--------+--------+
          |        |        |
          v        v        v
       Worker   Worker   Worker
          |        |        |
          +--------+--------+
                   |
                   v
              PostgreSQL

Gunicorn workers allow multiple requests to be processed concurrently.

But:

More workers does not automatically mean more performance.

If you create too many workers:

CPU
RAM
Database connections

can become bottlenecks.

Suppose each worker consumes significant memory.

If you launch 100 workers on a small server, you may create:

RAM exhaustion
        |
        v
Swapping
        |
        v
Everything becomes slower

Performance tuning is about balance.

Measure:

CPU utilization
memory usage
request latency
worker saturation
database connections

Then adjust.

17. Database Connections Can Become the Hidden Bottleneck

Imagine:

50 Django workers
       |
       v
50 database connections

Then you scale horizontally:

10 application instances
       |
       v
500 potential connections
       |
       v
PostgreSQL

Now your database becomes the bottleneck.

This is why horizontal scaling isn't simply:

More servers = more performance

A distributed system is a network of constraints.

You may have:

CPU
 |
 +--- Memory
 |
 +--- PostgreSQL
 |
 +--- Redis
 |
 +--- Network
 |
 +--- External APIs

The weakest component controls the system.

18. Use Connection Pooling Carefully

Opening database connections repeatedly has overhead.

Connection reuse can help.

Django supports persistent database connections through configuration such as:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "...",
        "USER": "...",
        "PASSWORD": "...",
        "HOST": "...",
        "PORT": "...",
        "CONN_MAX_AGE": 60,
    }
}

The exact configuration should depend on your deployment architecture.

At larger scale, dedicated connection pooling can become useful.

But again:

Don't add pooling because somebody on the internet said your application needs it.

Measure connection overhead and database saturation first.

19. Use `bulk_create()` Instead of Thousands of Inserts

Suppose you need to create 10,000 objects.

This:

for item in items:
    Product.objects.create(
        name=item["name"]
    )

can generate thousands of database operations.

Instead:

products = [
    Product(name=item["name"])
    for item in items
]

Product.objects.bulk_create(
    products,
    batch_size=1000
)

Now you can dramatically reduce database round trips.

The same principle applies to updates.

Instead of performing thousands of independent operations, look for opportunities to batch them.

Database performance is often about reducing trips.

20. Transactions Should Be Deliberate

Transactions are essential for correctness.

But long transactions can create contention.

Consider:

with transaction.atomic():
    do_expensive_calculation()
    call_external_api()
    update_database()

This is dangerous architecture.

You are potentially holding database resources while waiting for an external network request.

A better architecture is generally:

Calculate
   |
   v
External operation
   |
   v
Short database transaction
   |
   v
Commit

The goal is to keep critical database transactions as short as practical.

Performance and correctness aren't enemies.

Good transaction design improves both.

21. Optimize Templates Too

Django isn't only an API framework.

Traditional Django applications render templates.

Suppose you have:

{% for product in products %}
    {{ product.category.name }}
{% endfor %}

If the queryset isn't optimized, your template can indirectly trigger database queries.

The solution isn't to blame templates.

Fix the queryset:

products = Product.objects.select_related("category")

Templates should ideally operate on data that has already been prepared efficiently.

Think of rendering as the final stage:

Database
   |
   v
Optimized Queryset
   |
   v
Business Logic
   |
   v
Template
   |
   v
HTML

22. Static Files Should Not Be Served by Django

Your application server should focus on dynamic requests.

Things like:

CSS
JavaScript
Images
Fonts
Videos

should generally be served through appropriate static/media infrastructure.

A common architecture is:

                    Internet
                       |
                       v
                      CDN
                    /     \
                   /       \
              Static      Dynamic
                |            |
                v            v
             Assets       Django

If Django spends resources serving a 4 MB image, you're using expensive application capacity for a task better handled by a CDN or object-storage layer.

23. Compression Matters

If your API returns large JSON responses, compression can significantly reduce transfer size.

For example:

Uncompressed:
2.0 MB

Compressed:
300 KB

The exact ratio depends on the content.

This matters especially for:

mobile users
international users
high-latency networks
large API responses

Performance isn't only CPU.

It's also physics.

A byte has to travel from somewhere to somewhere else.

24. Use the Right Data Structure

Sometimes the database isn't the bottleneck.

Python itself can be.

For example:

users = []

for user in queryset:
    if user.id not in existing_ids:
        users.append(user)

If existing_ids is a list, membership checks can become increasingly expensive.

A set:

existing_ids = set(existing_ids)

can make membership checks dramatically cheaper.

This sounds small.

But imagine:

100,000 objects

inside nested loops.

Small algorithmic mistakes become large performance problems.

The difference between:

O(n)

and:

O(n²)

isn't theoretical when you're processing millions of records.

25. Move Heavy Computation Away From Requests

Suppose you're generating:

PDF reports
AI embeddings
video thumbnails
large CSV exports
machine-learning predictions
image processing
financial reports

Don't necessarily make the user wait for all of it.

Instead:

User
 |
 v
POST /reports
 |
 v
Django
 |
 +----> Create job
 |
 +----> Queue
          |
          v
        Worker
          |
          v
       Generate
          |
          v
       Storage

Then the frontend can poll:

GET /reports/123

or receive an event when the job is complete.

Now your web server stays responsive.

26. Profile Before and After

Suppose you've optimized your endpoint.

Don't say:

“It feels faster.”

Measure it.

Track:

p50
p95
p99

Why?

Because averages can lie.

Imagine:

Request 1: 50ms
Request 2: 50ms
Request 3: 50ms
Request 4: 5 seconds

The average hides the experience of the unlucky user.

Percentiles expose it.

For example:

p50 = 100ms
p95 = 500ms
p99 = 2.5s

This tells you that the tail of your system is unhealthy.

At scale, tail latency matters enormously.

27. A Practical Optimization Stack

If I inherited a slow Django application, I wouldn't randomly rewrite it.

I'd work through the system in layers.

              PERFORMANCE STACK

       +---------------------------+
       |       CDN / Browser       |
       +---------------------------+
       |      HTTP Compression     |
       +---------------------------+
       |       Nginx / Proxy       |
       +---------------------------+
       |    Gunicorn / ASGI        |
       +---------------------------+
       |          Django           |
       +---------------------------+
       |     Cache / Redis         |
       +---------------------------+
       |       PostgreSQL          |
       +---------------------------+
       |     Disk / Infrastructure |
       +---------------------------+

At each layer, ask:

HTTP

Are responses unnecessarily large?
Can content be cached?
Is compression enabled?

Django

Are views doing too much?
Are serializers causing queries?
Is middleware expensive?
Are synchronous operations blocking requests?

ORM

Are there N+1 queries?
Are querysets too large?
Are relationships prefetched?
Are only required fields selected?

PostgreSQL

Are indexes correct?
Are queries using them?
What does EXPLAIN ANALYZE say?
Are transactions too long?

Cache

What can be cached?
What is the cache hit rate?
Are cache keys correct?
How is invalidation handled?

Infrastructure

Are workers saturated?
Is memory exhausted?
Is the database connection limit being reached?
Is the network the bottleneck?

This is how you turn "Django is slow" into an engineering problem.

28. An Example: Turning a Slow Endpoint Into a Fast Endpoint

Imagine this endpoint:

def orders(request):
    orders = Order.objects.all()

    result = []

    for order in orders:
        result.append({
            "id": order.id,
            "customer": order.customer.name,
            "items": [
                {
                    "name": item.product.name,
                    "price": item.price,
                }
                for item in order.items.all()
            ]
        })

    return JsonResponse({
        "orders": result
    })

This endpoint has several problems.

It loads every order.

It accesses customers lazily.

It accesses items lazily.

It accesses products lazily.

It returns everything.

Let's redesign it.

def orders(request):
    orders = (
        Order.objects
        .select_related("customer")
        .prefetch_related("items__product")
        .order_by("-created_at")[:50]
    )

    result = []

    for order in orders:
        result.append({
            "id": order.id,
            "customer": order.customer.name,
            "items": [
                {
                    "name": item.product.name,
                    "price": item.price,
                }
                for item in order.items.all()
            ]
        })

    return JsonResponse({
        "orders": result
    })

Now the architecture is much better.

We have:

Before:

All orders
   |
   +--> Customer query
   +--> Customer query
   +--> Customer query
   |
   +--> Item query
   +--> Item query
   +--> Item query
   |
   +--> Product query
   +--> Product query
   +--> Product query
   |
   v
Huge response


After:

50 orders
   |
   +--> select_related(customer)
   |
   +--> prefetch_related(items__product)
   |
   v
Small response

The important thing is that we didn't rewrite Django.

We simply stopped asking it to perform unnecessary work.

29. The 10x Formula

There is no universal switch called:

DJANGO_10X = True

Real performance improvements usually come from multiplying improvements across layers.

For example:

2x from query optimization
×
2x from caching
×
1.5x from reducing serialization
×
1.5x from smaller responses
×
1.2x from infrastructure tuning

≈ 10.8x

These improvements don't always multiply this cleanly in reality, but the concept is important.

You don't necessarily need one miraculous optimization.

You need several high-leverage optimizations.

30. The Most Important Performance Skill

The most valuable Django performance skill isn't memorizing:

select_related()

or:

prefetch_related()

or:

cache.get()

It's learning to ask:

Where is the time going?

Suppose an endpoint takes:

1.2 seconds

Break it down:

Network:          50ms
Django:           100ms
Database:         800ms
Serialization:    150ms
External API:     100ms

Now you know what to attack.

The database is responsible for most of the latency.

So spend your time there.

After optimization:

Network:           50ms
Django:             80ms
Database:          120ms
Serialization:      50ms
External API:      100ms

Total:             400ms

You just cut latency by roughly two-thirds.

That's engineering.

31. Don't Optimize Everything

This is where experienced engineers differ from beginners.

Beginners often want to optimize everything.

Experts optimize the bottleneck.

If a function takes:

0.2 ms

and you make it:

0.1 ms

you probably achieved nothing meaningful.

If a query takes:

900 ms

and you make it:

50 ms

you changed the system.

Optimization should follow impact.

A useful priority system is:

             High Impact
                  ^
                  |
       Database   |   External APIs
                  |
       Caching    |
                  |
       ORM        |
                  |
------------------+------------------> Effort
                  |
       Python     |
       micro-opt  |
                  |
             Low Impact

Attack the upper-left region first.

32. A Performance Checklist

When optimizing Django, I mentally run through something like this:

[ ] Measure endpoint latency
[ ] Measure p50/p95/p99
[ ] Count database queries
[ ] Find N+1 queries
[ ] Inspect slow SQL
[ ] Run EXPLAIN ANALYZE
[ ] Add appropriate indexes
[ ] Use select_related()
[ ] Use prefetch_related()
[ ] Select only required fields
[ ] Use exists() for existence checks
[ ] Use count() when counting
[ ] Aggregate in the database
[ ] Paginate large datasets
[ ] Batch writes
[ ] Cache expensive reads
[ ] Reduce response size
[ ] Compress responses
[ ] Serve static assets efficiently
[ ] Move long jobs to workers
[ ] Review external API calls
[ ] Review database connections
[ ] Tune worker counts
[ ] Monitor CPU and memory
[ ] Measure again

The last item is the most important.

Measure again.

33. The Architecture of a Fast Django System

A mature Django application doesn't look like:

Browser
   |
   v
Django
   |
   v
Database

It starts looking more like:

                         INTERNET
                            |
                            v
                           CDN
                            |
                            v
                         NGINX
                            |
                    +-------+-------+
                    |               |
                    v               v
                 Static          Django
                                   |
                    +--------------+--------------+
                    |              |              |
                    v              v              v
                  Redis        PostgreSQL       Queue
                                                   |
                                                   v
                                                 Workers
                                                   |
                                                   v
                                           External Services

The architecture works because each component has a job.

Django handles application logic.

PostgreSQL handles relational data.

Redis handles fast temporary state and caching.

Workers handle long-running tasks.

CDNs handle globally distributed static content.

Nginx handles reverse-proxy concerns.

The result isn't merely "faster Django."

It's less work for Django.

And that's the secret.

34. Django Isn't Usually the Problem

This is probably the most controversial thing I'll say:

Django is often blamed for performance problems that Django didn't create.

The ORM didn't decide to create 1,001 queries.

Your application did.

Django didn't decide to return 50 MB of JSON.

Your API did.

Django didn't decide to execute an external API call synchronously.

Your architecture did.

Django didn't forget your database index.

Your schema did.

Django didn't decide to process 500,000 objects in Python.

Your code did.

Frameworks give you tools.

They don't automatically give you good architecture.

And this is why the difference between a slow Django application and a fast Django application can be enormous even when both use exactly the same framework.

35. The Real Secret to 10x Performance

Making Django 10x faster isn't about discovering a secret Django setting.

It is about changing the amount of work your system performs.

Think about the progression:

Bad system:

Request
  |
  v
Django
  |
  +--> Huge query
  |
  +--> N+1 queries
  |
  +--> Expensive computation
  |
  +--> External API
  |
  +--> Giant serialization
  |
  +--> Giant response
  |
  v
Client

Now compare that with:

Optimized system:

Request
  |
  v
Cache ---- HIT ----> Response
  |
 MISS
  |
  v
Django
  |
  +--> Optimized query
  |
  +--> Indexed database
  |
  +--> Small dataset
  |
  +--> Efficient serialization
  |
  +--> Background jobs
  |
  v
Compressed response

The second system isn't necessarily running Python 10 times faster.

It's doing dramatically less unnecessary work.

That's a much more powerful optimization.

Conclusion: Make Django Boringly Fast

I like fast systems.

But I like predictable systems even more.

A backend that randomly takes 50 ms sometimes and 8 seconds at other times isn't truly fast.

A good production system should have:

Predictable latency
+
Efficient queries
+
Controlled memory
+
Healthy database connections
+
Effective caching
+
Small responses
+
Asynchronous background work
+
Good observability

And Django can absolutely be part of that architecture.

The path to making Django 10x faster is usually not:

Rewrite everything.

It's:

Measure everything.

Then:

Find the bottleneck.

Then:

Remove unnecessary work.

Then:

Measure again.

Find the N+1 query.

Remove it.

Find the missing index.

Add it.

Find the 5 MB response.

Shrink it.

Find the repeated calculation.

Cache it.

Find the 30-second operation.

Move it to a worker.

Find the unnecessary database call.

Delete it.

Find the endpoint waiting on three APIs sequentially.

Make the architecture concurrent or asynchronous where appropriate.

Then measure again.

That's performance engineering.

And once you start seeing applications this way, something interesting happens.

You stop asking:

“How do I make Django faster?”

You start asking the much more powerful question:

“Why is the computer doing this work at all?”

That question is where the real 10x improvements begin.

How I Went From Junior to Senior Developer

Derek Mwale — Tue, 25 Aug 2026 09:36:12 +0000

There was a point in my career when I thought becoming a senior developer meant learning more programming languages.

I thought senior engineers knew everything.

They could look at a codebase and immediately understand it.

They could write complicated algorithms from memory.

They knew every framework.

They could solve bugs nobody else could solve.

They had some mysterious internal database containing the entire documentation of the internet.

So I tried to become one.

I learned frameworks.

I learned languages.

I built projects.

I read documentation.

I watched tutorials.

I wrote code late at night.

And slowly, something changed.

But it wasn't what I expected.

I didn't become senior because I learned everything.

I became more senior when I realized I would never know everything.

The real transition from junior to senior wasn't about knowing more syntax.

It was about learning how to think.

And that changed everything.

The Junior Developer Trap

When you're junior, programming can feel like a collection of puzzles.

You receive a requirement.

You write code.

The code works.

You move on.

Requirement
    ↓
Code
    ↓
Works
    ↓
Next Task

That is a perfectly reasonable way to start.

You are learning.

You need repetition.

You need to build things.

You need to make mistakes.

You need to experience what happens when your beautiful code meets reality.

But eventually, the questions change.

A junior developer might ask:

“How do I implement this feature?”

A more experienced developer starts asking:

“Should we implement this feature this way?”

That is a massive difference.

The first question is about implementation.

The second is about engineering.

I Stopped Thinking About Code as the Product

One of the biggest lessons I learned was that code isn't the product.

Software is the product.

And those are not the same thing.

You can write beautiful code that solves the wrong problem.

You can write clever code that nobody can maintain.

You can build an elegant abstraction around something that should have been simple.

You can optimize a function that doesn't matter.

You can build a technically impressive system that nobody wants to use.

Eventually, I started thinking about software as a system.

Users
  ↓
Requirements
  ↓
Business Rules
  ↓
Architecture
  ↓
Code
  ↓
Infrastructure
  ↓
Operations

Code is only one layer.

Once I understood that, my relationship with programming changed.

I stopped asking:

“How can I write this?”

And started asking:

“What is the simplest system that solves this correctly?”

That question is much harder.

It is also much more valuable.

I Learned to Read Code Before Writing Code

Junior developers often want to immediately start coding.

I understand why.

Coding feels productive.

You open the IDE.

You create a file.

You start typing.

Suddenly you feel like you're making progress.

But in an existing system, writing code too early can be dangerous.

Before changing anything, I learned to investigate.

I ask:

Where does the request enter?

Where is authentication handled?

Where is the business logic?

Where is the database accessed?

Where are errors handled?

Where are tests?

What services depend on this code?

What assumptions already exist?

Then I trace the system.

For a backend feature, for example:

HTTP Request
     ↓
Router
     ↓
Controller
     ↓
Service
     ↓
Repository
     ↓
Database

I want to understand that flow before modifying it.

This sounds slower.

It isn't.

Understanding the system first often saves hours of debugging later.

I Became Comfortable Being Confused

This was another major transition.

Junior developers sometimes interpret confusion as failure.

I used to think:

“If I'm good at programming, I should understand this immediately.”

Not true.

Complex systems are complex.

Distributed systems are complex.

Databases are complex.

Operating systems are complex.

Networks are complex.

Legacy codebases are sometimes archaeological sites.

Nobody understands everything immediately.

Senior engineers aren't people who never get confused.

They are people who know how to reduce confusion.

That distinction matters.

When something doesn't make sense, I break it down.

Big Problem
    ↓
Smaller Problem
    ↓
Smaller Problem
    ↓
Unknown
    ↓
Experiment
    ↓
Evidence
    ↓
Understanding

Instead of saying:

“I don't understand the system.”

I ask:

“Which specific part don't I understand?”

Then:

“What evidence can I collect?”

That mindset changed how I debug.

Debugging Became a Scientific Process

Early in my career, debugging sometimes looked like this:

Error
 ↓
Google
 ↓
Copy solution
 ↓
Try
 ↓
Another error
 ↓
Copy another solution

We've all done it.

But eventually I realized something.

Debugging isn't primarily about finding code snippets.

It's about forming hypotheses.

Suppose an API returns HTTP 500.

Instead of randomly changing things:

Maybe database?
Maybe serializer?
Maybe authentication?
Maybe Docker?
Maybe network?

I create hypotheses.

H1: Database query is failing.
H2: Input validation is failing.
H3: External service is unavailable.
H4: Application state is invalid.

Then I collect evidence.

Logs
Metrics
Stack traces
Database state
Request payload
Network response

Then I eliminate possibilities.

That is engineering.

I Learned to Read Error Messages Properly

One of the simplest changes with the biggest impact was learning to actually read errors.

Not just the final line.

The entire stack trace.

Suppose I see:

AttributeError:
'int' object has no attribute 'strip'

The useful information isn't merely:

Something crashed.

It is:

An integer reached code expecting a string.

Now I can ask:

Where did the value originate?

Why did the type change?

Why wasn't it validated?

Which layer allowed it through?

The error becomes evidence.

Senior engineering is often about turning symptoms into causes.

I Started Thinking About Failure First

Junior development often focuses on the happy path.

User
 ↓
Request
 ↓
Success

Production doesn't work like that.

Production looks like:

Request
 ├── valid
 ├── invalid
 ├── duplicated
 ├── delayed
 ├── malicious
 ├── partially completed
 ├── retried
 └── interrupted

So I started designing for failure.

What happens if the database is down?

What happens if the API times out?

What happens if the user submits twice?

What happens if the worker crashes halfway through?

What happens if a payment succeeds but our server crashes before recording it?

What happens if the queue receives the same message twice?

These questions completely change architecture.

For example:

Payment Request
      ↓
Create Idempotency Key
      ↓
Process Payment
      ↓
Persist Result
      ↓
Return Response

Now a retry doesn't necessarily create another payment.

That's not advanced syntax.

It's advanced thinking.

I Stopped Overengineering

Ironically, becoming more experienced also taught me to write less code.

When I was younger, complexity felt impressive.

Multiple abstractions.

Design patterns everywhere.

Generic systems.

Highly configurable components.

Layers upon layers.

Then I discovered something painful:

Complexity has a maintenance cost.

Every abstraction creates another thing future developers must understand.

Every configuration option creates another state.

Every dependency creates another failure mode.

Eventually I started asking:

“Does this abstraction solve a real problem?”

If not, remove it.

A simple function is often better than a framework built around a function.

A PostgreSQL query is sometimes better than a complicated repository abstraction.

A boring service can be better than an elaborate microservice architecture.

Senior engineering isn't about maximizing sophistication.

It's about minimizing unnecessary complexity.

I Learned That “It Works” Is Not Enough

This was one of the biggest mindset changes.

Imagine this API:

@app.post("/users")
def create_user(data):
    user = User.objects.create(
        name=data["name"],
        email=data["email"]
    )

    return user

It works.

But what about:

Invalid email?
Duplicate email?
Missing fields?
Unauthorized request?
Race condition?
Database failure?
Sensitive data?
Logging?
Rate limiting?
Transaction boundaries?

A senior mindset asks about the system around the code.

The real question isn't:

“Does it work?”

It is:

“Under what conditions does it work?”

And:

“What happens when those conditions aren't true?”

That is the difference between a demo and production software.

I Started Thinking in Trade-Offs

This is probably one of the clearest differences between junior and senior engineering.

Junior thinking often looks for the correct answer.

Senior engineering often involves choosing between imperfect answers.

For example:

PostgreSQL vs MongoDB
REST vs GraphQL
Monolith vs Microservices
Sync vs Async
Cache vs Fresh Data
Consistency vs Availability
Speed vs Maintainability
Cost vs Performance

There is rarely a universal winner.

The question becomes:

“What are the trade-offs for this system?”

Suppose a monolith is easier to deploy and maintain.

A microservice architecture may provide stronger service isolation.

But now you introduce:

Network calls
Service discovery
Distributed tracing
Deployment complexity
Data consistency problems
More infrastructure

So I stopped asking:

“Which architecture is best?”

I started asking:

“Which architecture is appropriate?”

That is a much better question.

I Learned Databases More Seriously

Another major turning point was realizing that backend development isn't really about APIs.

It's about data.

The API is just one interface to the data and business logic.

I started caring more about:

Indexes
Transactions
Constraints
Isolation
Query plans
Locks
Normalization
Denormalization
Connection pools
Migrations
Consistency

Suppose you have:

SELECT *
FROM orders
WHERE user_id = 42;

A junior developer might think:

“The query works.”

A more experienced developer asks:

How many rows exist?

Is user_id indexed?

What does EXPLAIN say?

How frequently is this query executed?

What happens at 10 million rows?

Is SELECT * necessary?

What happens under concurrent writes?

That's database engineering.

And databases teach you something beautiful:

Software eventually meets mathematics.

I Learned That APIs Are Contracts

An API isn't merely an endpoint.

It's a contract between systems.

Suppose you have:

POST /api/orders

A mature API needs to define:

Input
Output
Authentication
Authorization
Errors
Validation
Idempotency
Pagination
Versioning
Rate limits

For example:

{
  "product_id": 42,
  "quantity": 3
}

The important question isn't simply:

“Can I accept this JSON?”

It is:

“What guarantees does this endpoint make?”

Once I started thinking about APIs as contracts, I became much more careful about backwards compatibility.

Changing:

{
  "name": "Derek"
}

into:

{
  "full_name": "Derek Mwale"
}

isn't merely renaming a variable.

It can break clients.

APIs live longer than the code that created them.

I Learned to Write for the Next Developer

There is a famous temptation in programming:

“I know what this code does, so it is obvious.”

Six months later, you become the next developer.

And suddenly:

x = process(data)

looks like archaeology.

Good engineering communicates intent.

Sometimes that means better naming:

eligible_customers = get_eligible_customers()

instead of:

x = get_data()

Sometimes it means a comment explaining why something strange exists.

Not:

# increment i
i += 1

But:

# We intentionally process records in batches because the
# external provider rejects requests larger than 100 items.

The best comments explain decisions.

Code explains mechanics.

I Started Reviewing My Own Code

Code review isn't just something other people do to you.

Eventually, I started reviewing my own code before opening a pull request.

I ask:

Would I approve this?

Is this readable?

Is there unnecessary complexity?

What happens on failure?

Are errors handled correctly?

Are tests meaningful?

Is the API consistent?

Could this create a security problem?

Could this query become expensive?

What would happen at 10x scale?

That last question is particularly useful.

Not because every application needs to scale to millions of users.

But because scaling questions reveal architectural assumptions.

I Learned Testing Is About Confidence

Early testing can feel like bureaucracy.

Write code.

Write tests.

Fix tests.

Repeat.

But testing eventually becomes freedom.

Suppose I refactor a payment service.

Without tests:

Change
 ↓
Hope
 ↓
Deploy
 ↓
Discover

With tests:

Change
 ↓
Test
 ↓
Confidence
 ↓
Deploy

The most important thing isn't 100% coverage.

It's meaningful coverage.

Test business rules.

Test failure conditions.

Test boundaries.

Test contracts.

For example:

def test_duplicate_payment_is_idempotent():
    first = create_payment(
        idempotency_key="abc"
    )

    second = create_payment(
        idempotency_key="abc"
    )

    assert first.id == second.id

That test encodes a business guarantee.

That's valuable.

I Started Learning the Infrastructure Beneath the Code

At some point, “backend developer” stopped meaning:

“I know Django.”

It started meaning:

“I understand what happens after the request leaves the browser.”

That includes:

DNS
HTTP
TLS
Load Balancers
Reverse Proxies
Containers
Linux
Processes
Networking
Databases
Queues
Caching
Cloud Infrastructure

Understanding these layers makes debugging much easier.

If an API is slow, the answer isn't automatically:

“The Python code is slow.”

It might be:

DNS
 ↓
TLS handshake
 ↓
Load balancer
 ↓
Application
 ↓
Database
 ↓
External API

The bottleneck could exist anywhere.

The more layers you understand, the better you can reason about the system.

I Learned Git Is Part of Engineering

Git isn't just:

git add .
git commit
git push

It is part of collaboration.

Good commits communicate intent.

Bad:

fix stuff
changes
update
final
final2

Better:

Add idempotency protection to payment creation

Now the history tells a story.

Version control is not only about recovering old files.

It's about understanding how a system evolved.

I Became Better at Asking Questions

This might be one of the most underrated senior skills.

A junior developer sometimes asks:

“How do I do this?”

A stronger question is:

“I think the problem is X because of Y. I considered approaches A and B. I'm leaning toward A because of Z. Does that match your understanding?”

That communicates:

Investigation
Reasoning
Ownership
Communication

It also makes collaboration easier.

The goal isn't to prove that you don't need help.

The goal is to make asking for help efficient.

I Learned That Communication Is a Technical Skill

You can be an incredible programmer and still be difficult to work with.

Software development is collaborative.

You need to communicate:

What happened?
Why did it happen?
What are the options?
What are the risks?
What do you recommend?
What do you need?

A strong engineer can explain a complicated technical problem to:

Another engineer
Product manager
Designer
Founder
Customer
Non-technical stakeholder

without turning the conversation into a lecture about implementation details.

If you can build something but cannot explain why it should exist, you have only mastered part of engineering.

I Started Thinking About Business

This was another major shift.

Software doesn't exist in isolation.

A technically perfect system that solves no meaningful problem is not a successful product.

So I started asking:

Who uses this?

Why do they care?

What problem does it solve?

What does failure cost?

What does success look like?

What is the simplest version?

What should we not build?

A senior engineer doesn't need to become a product manager.

But they should understand the product.

Because architecture is downstream from requirements.

I Learned to Say No

This is surprisingly difficult.

Sometimes the correct technical decision is:

“We shouldn't build that.”

Not because the feature is bad.

Because the cost is greater than the value.

Maybe the system doesn't need microservices.

Maybe it doesn't need Kubernetes.

Maybe it doesn't need an AI agent.

Maybe it doesn't need five databases.

Maybe it doesn't need a custom framework.

Maybe it doesn't need another abstraction.

Engineering is partly the art of deciding what not to build.

I Stopped Chasing Every New Technology

Technology moves incredibly fast.

New framework.

New database.

New programming language.

New AI model.

New cloud platform.

New architecture pattern.

If you chase everything, you never build depth.

I started focusing on fundamentals.

Programming
 ↓
Data Structures
 ↓
Algorithms
 ↓
Databases
 ↓
Networking
 ↓
Operating Systems
 ↓
Distributed Systems
 ↓
Architecture

Frameworks change.

Fundamentals remain.

Django can change.

React can change.

Cloud providers can change.

AI models can change.

But:

Latency
Consistency
Concurrency
Transactions
Memory
Networking
Complexity

aren't going anywhere.

I Learned to Build Instead of Just Learn

Tutorials are useful.

Courses are useful.

Documentation is essential.

But eventually you have to build something that can break.

That's where learning becomes real.

Build:

An API
A database engine
A queue
A CLI
A compiler
A game
A search engine
A distributed service
An authentication system
An AI application

Then make it fail.

Because building gives you questions that tutorials don't.

Why is this slow?

Why did this race condition happen?

Why did the database lock?

Why did Docker fail?

Why did the queue duplicate the message?

Why did memory usage explode?

Those questions create engineers.

The Senior Developer Is Not the Fastest Coder

This is probably the biggest misconception I had.

Senior developers are not necessarily the people who type the fastest.

They are often the people who prevent the team from building the wrong thing.

A junior might write:

500 lines of code

in a day.

A senior might spend a day asking questions and then write:

100 lines.

And those 100 lines might solve the problem better.

The value isn't measured by keyboard activity.

It's measured by outcomes.

My Definition of Senior Changed

Today, I don't define seniority as:

Years of experience
+
Number of programming languages
+
Framework knowledge

I think of it more like:

Senior Engineering
=
Technical Depth
+
System Thinking
+
Decision Making
+
Communication
+
Ownership
+
Judgment

Technical knowledge matters.

But judgment is what ties it together.

Knowing ten ways to solve a problem is useful.

Knowing which one to choose is more valuable.

The Junior-to-Senior Transition

If I had to visualize the journey, it would look something like this:

                 JUNIOR
                   │
                   ▼
             Learn Syntax
                   │
                   ▼
             Build Features
                   │
                   ▼
             Debug Problems
                   │
                   ▼
          Understand Systems
                   │
                   ▼
          Understand Trade-offs
                   │
                   ▼
          Design Solutions
                   │
                   ▼
          Own Technical Decisions
                   │
                   ▼
            Help Others Grow
                   │
                   ▼
                 SENIOR

But there is something interesting here.

The progression isn't:

Junior → Knows Everything → Senior

It is:

Junior → Learns How to Code
        ↓
Developer → Learns How Systems Work
        ↓
Senior → Learns How to Make Good Decisions

And even that isn't the end.

Because after senior comes another realization:

You don't become valuable by being the person who knows everything.

You become valuable by making the entire team better at solving problems.

What I Would Tell My Junior Self

If I could go back and talk to the version of myself who was trying to become a better developer, I wouldn't tell him to memorize more syntax.

I'd tell him:

Learn the fundamentals.

Build real things.

Read code written by other people.

Read error messages.

Understand databases.

Understand networking.

Learn Linux.

Learn Git properly.

Write tests.

Learn how APIs actually work.

Study system design.

Don't fear difficult problems.

Don't pretend to understand something when you don't.

Ask better questions.

Don't overengineer.

Don't blindly copy architecture from companies operating at a scale you don't have.

Understand trade-offs.

Learn to communicate.

Take ownership.

And most importantly:

Don't measure your growth by how much code you can write.

Measure it by how much complexity you can understand.

Final Thoughts

Going from junior to senior wasn't one dramatic event.

There was no morning where I woke up and suddenly received the mythical senior developer powers.

It happened gradually.

One bug at a time.

One architecture decision at a time.

One failed deployment at a time.

One database problem at a time.

One code review at a time.

One uncomfortable technical conversation at a time.

I learned that programming is not really about writing instructions for computers.

It's about understanding problems deeply enough to design systems that survive contact with reality.

The junior developer asks:

“How do I make this work?”

The senior developer asks:

“What should this system guarantee?”

And eventually, an even better question appears:

“What is the simplest way to make those guarantees true?”

That is the shift.

From syntax to systems.

From implementation to judgment.

From writing code to designing software.

From solving your own problems to helping an entire team solve theirs.

I didn't become senior because I stopped making mistakes.

I became more senior because I became better at understanding why the mistakes happened, how to prevent them, and how to turn them into engineering knowledge.

And perhaps that is what seniority really is.

Not knowing everything.

But knowing how to think when you don't.

Building Semantic Search: From Embeddings to a Production-Ready Search Engine

Derek Mwale — Tue, 25 Aug 2026 09:07:03 +0000

Search looks simple until you try to build a system that understands what people actually mean.

A traditional search engine sees:

"cheap laptops for developers"

and starts looking for documents containing:

cheap
laptops
developers

That works surprisingly well.

Until the user searches for:

"affordable machines for coding"

Now the problem becomes obvious.

The second query may be semantically almost identical to the first one, but the words are different.

A keyword search engine sees two different strings.

A semantic search engine sees two related ideas.

That difference is the foundation of modern intelligent search.

Semantic search attempts to answer a more interesting question:

Which documents are conceptually closest to what the user is asking for?

Instead of treating text as a collection of isolated words, we transform text into mathematical representations called embeddings.

Those embeddings allow us to represent meaning as vectors.

Then search becomes a geometric problem.

                    Text
                     │
                     ▼
              Embedding Model
                     │
                     ▼
                Vector
                     │
                     ▼
          ┌────────────────────┐
          │ Vector Database    │
          │                    │
          │ Similarity Search  │
          └─────────┬──────────┘
                    │
                    ▼
             Relevant Results

This article explores how to build semantic search from first principles and then turn it into something that looks much closer to a production backend system.

We will cover:

embeddings
vector spaces
cosine similarity
Euclidean distance
nearest-neighbor search
PostgreSQL + pgvector
document chunking
embedding pipelines
metadata filtering
hybrid search
reranking
indexing
FastAPI implementation
database schema design
query optimization
caching
evaluation
failure modes
production architecture
security
and the deeper engineering principles behind semantic search.

The important part is not simply getting a vector database working.

The important part is understanding why semantic search works and where it fails.

1. The Problem With Keyword Search

Let's begin with a conventional search system.

Imagine a database containing articles:

1. Building APIs with Django
2. PostgreSQL indexing strategies
3. Machine learning for agriculture
4. Designing distributed systems
5. Securing REST APIs

A user searches:

"How do I protect my backend endpoints?"

A keyword engine might search for:

protect
backend
endpoints

But the document might say:

"Authentication and authorization are fundamental
components of secure API architecture."

There may be almost no exact word overlap.

Yet the document is clearly relevant.

This gives us the first fundamental distinction:

Keyword Search

query words
    │
    ▼
exact / lexical matching
    │
    ▼
relevant text


Semantic Search

query meaning
    │
    ▼
vector representation
    │
    ▼
semantic similarity
    │
    ▼
relevant concepts

Keyword search asks:

"Which documents contain these words?"

Semantic search asks:

"Which documents mean something similar to this query?"

Neither is universally superior.

The strongest production search engines often combine both.

But before we combine them, we need to understand the mathematics.

2. What Is an Embedding?

An embedding is a numerical representation of some object.

For text, an embedding model converts a piece of text into a vector.

For example, imagine our model produces only three dimensions:

"backend development"

→ [0.82, 0.14, 0.61]

And:

"server-side programming"

→ [0.79, 0.17, 0.64]

While:

"banana farming"

→ [0.08, 0.91, 0.12]

The first two vectors are close together.

The third is far away.

                    farming
                       ●
                      /
                     /
                    /
                   /
                  /
       ● backend
      /
     ● server-side

The embedding model is effectively transforming language into geometry.

That is one of the most important ideas in modern machine learning.

3. Semantic Search Is Geometry

Suppose an embedding model generates vectors in:

[
\mathbb{R}^n
]

That means every document becomes a point in an (n)-dimensional space.

We cannot visualize thousands of dimensions directly.

But conceptually:

                    Dimension 2
                         ↑
                         │
                 ●       │
                         │
                         │
          ●              │
                         │
                         └────────────────→ Dimension 1

Similar concepts tend to occupy nearby regions of the embedding space.

The search problem becomes:

[
q = embedding(query)
]

Then:

[
d_i = embedding(document_i)
]

We want:

[
d_i \approx q
]

according to some similarity function.

This means semantic search is fundamentally:

Text
 │
 ▼
Embedding
 │
 ▼
Vector
 │
 ▼
Distance / Similarity
 │
 ▼
Nearest neighbors
 │
 ▼
Results

That sounds simple.

The engineering becomes interesting when we have millions of vectors.

4. Why Embeddings Capture Meaning

An embedding model is trained to produce useful numerical representations.

It doesn't literally store a dictionary where:

"cat" = [....]

and:

"dog" = [....]

Instead, the model learns statistical relationships between language concepts.

As a result, related concepts can occupy similar regions of the vector space.

For example:

"software engineer"
"backend developer"
"programmer"
"application developer"

may have representations that are closer to each other than to:

"mountain climbing"

This creates a semantic topology.

             Programming
                  ●
              ●       ●
        backend       frontend
             ●         ●
               ● API



                                    Agriculture
                                         ●
                                     ●       ●
                                 farming    crops

The embedding model is effectively creating a coordinate system for meaning.

But there is an important caveat:

The coordinates themselves do not have human-readable meanings.

Dimension 42 does not necessarily mean "technicality."

Dimension 100 does not necessarily mean "agriculture."

The representation is distributed.

Meaning emerges from relationships between vectors.

5. The First Mathematical Primitive: Dot Product

Suppose we have two vectors:

[
A = [a_1, a_2, ..., a_n]
]

and:

[
B = [b_1, b_2, ..., b_n]
]

Their dot product is:

[
A \cdot B =
\sum_{i=1}^{n} a_i b_i
]

For example:

[
A = [1,2,3]
]

[
B = [4,5,6]
]

Then:

[
A \cdot B =
1(4)+2(5)+3(6)
]

[
=4+10+18
]

[
=32
]

The dot product can provide a measure of alignment.

But vector magnitude matters.

That leads us to cosine similarity.

6. Cosine Similarity

Cosine similarity measures the angle between two vectors.

The formula is:

[

cos(\theta)

\frac{A \cdot B}
{|A||B|}
]

where:

[

|A|

\sqrt{\sum a_i^2}
]

The important idea is that cosine similarity cares primarily about direction, not absolute magnitude.

Visualize it:

                 B
                /
               /
              /
             /
            / θ
           /
          /
         ●──────────────► A

If the vectors point in almost the same direction:

cos(θ) ≈ 1

If they are perpendicular:

cos(θ) ≈ 0

If they point in opposite directions:

cos(θ) ≈ -1

For many embedding systems, cosine similarity is a useful similarity measure.

7. Implementing Cosine Similarity

Python makes this easy:

import math


def cosine_similarity(a, b):
    dot = sum(
        x * y
        for x, y in zip(a, b)
    )

    magnitude_a = math.sqrt(
        sum(x * x for x in a)
    )

    magnitude_b = math.sqrt(
        sum(x * x for x in b)
    )

    if magnitude_a == 0 or magnitude_b == 0:
        return 0.0

    return dot / (
        magnitude_a * magnitude_b
    )

Now:

a = [1, 2, 3]
b = [2, 4, 6]

print(cosine_similarity(a, b))

The result is approximately:

1.0

because the vectors point in exactly the same direction.

This tiny function demonstrates the mathematical core of many semantic search systems.

But it does not scale.

8. The Naive Search Algorithm

Suppose we have:

1 million documents

Each has an embedding.

A naive search does:

query
  │
  ▼
embedding
  │
  ▼
compare against document 1
  │
  ▼
compare against document 2
  │
  ▼
compare against document 3
  │
  ▼
...
  │
  ▼
compare against document 1,000,000
  │
  ▼
sort
  │
  ▼
top 10

Mathematically, this is approximately:

[
O(ND)
]

where:

(N) = number of vectors
(D) = vector dimension

For small datasets, this can be perfectly acceptable.

For large datasets, it becomes expensive.

This is where approximate nearest-neighbor indexing enters the picture.

9. Exact Search vs Approximate Search

There are two broad approaches.

Exact nearest-neighbor search

Compare the query against every vector.

Advantages:

exact
simple
predictable

Disadvantages:

expensive at scale

Approximate nearest-neighbor search

Use an index structure to avoid comparing against every vector.

Advantages:

dramatically faster
scales to large datasets

Disadvantages:

results are approximate
index tuning matters

The architecture becomes:

Exact Search

Query
 │
 ├── Vector 1
 ├── Vector 2
 ├── Vector 3
 ├── Vector 4
 ├── ...
 └── Vector N
 │
 ▼
Top K


Approximate Search

Query
 │
 ▼
Vector Index
 │
 ├── candidate region
 ├── candidate region
 └── candidate region
 │
 ▼
Top K

This is one of the key engineering tradeoffs in semantic search.

10. PostgreSQL Can Become a Vector Database

If you're already building backend systems with PostgreSQL, you don't necessarily need to introduce a separate vector database immediately.

The pgvector extension allows PostgreSQL to store and search vector embeddings.

Conceptually:

                    PostgreSQL
                         │
       ┌─────────────────┼─────────────────┐
       │                 │                 │
       ▼                 ▼                 ▼
   relational         JSONB             vectors
      data                                │
                                          ▼
                                   semantic search

This is particularly attractive for applications where:

documents already live in PostgreSQL
users already live in PostgreSQL
permissions already live in PostgreSQL
metadata already lives in PostgreSQL

Instead of:

PostgreSQL
+
Vector Database
+
synchronization

you can initially use:

PostgreSQL
    │
    ├── relational records
    ├── metadata
    └── embeddings

Fewer systems means fewer synchronization problems.

11. Designing the Database Schema

Let's build a document table.

CREATE EXTENSION IF NOT EXISTS vector;

Then:

CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,

    title TEXT NOT NULL,

    content TEXT NOT NULL,

    embedding VECTOR(1536),

    metadata JSONB NOT NULL DEFAULT '{}',

    created_at TIMESTAMPTZ
        NOT NULL DEFAULT now()
);

The 1536 dimension here is only an example.

Your vector dimension must match the embedding model you actually use.

You can add metadata such as:

{
  "category": "technology",
  "author": "Derek",
  "language": "en",
  "visibility": "public"
}

This is important because semantic similarity alone is rarely enough.

12. Documents Should Usually Be Chunked

This is one of the most important practical decisions.

Suppose you have a 10,000-word article.

You could generate one embedding:

10,000-word article
        │
        ▼
     embedding

But now the vector represents the entire document.

A user might ask:

"How does the article explain PostgreSQL indexing?"

The answer could be buried in one paragraph.

The whole-document embedding may dilute that specific concept.

Instead, split the document into chunks.

Article
 │
 ├── Chunk 1
 ├── Chunk 2
 ├── Chunk 3
 ├── Chunk 4
 ├── Chunk 5
 └── Chunk 6

Each chunk gets its own embedding.

Chunk 1 ──► Vector 1
Chunk 2 ──► Vector 2
Chunk 3 ──► Vector 3
...
Chunk 6 ──► Vector 6

Now search can identify the most relevant passage.

13. Chunking Is a Retrieval Problem

A common beginner mistake is:

"Split every 500 characters."

That's not necessarily good.

Imagine:

Chunk 1:
PostgreSQL indexes improve query performance...

Chunk 2:
...but only when the index matches the access pattern.

The second sentence depends on the first.

If we split them badly, we lose context.

Better chunking often respects:

paragraphs
headings
sentences
code blocks
lists
semantic sections

For technical documents, structure-aware chunking can be especially valuable.

A document might become:

Article
│
├── Introduction
│
├── What is PostgreSQL?
│
├── Indexing
│   ├── B-tree indexes
│   ├── Partial indexes
│   └── Composite indexes
│
├── Transactions
│
└── Conclusion

This structure gives us much better retrieval units.

14. Chunk Size Is a Tradeoff

Very small chunks:

Pros:
- precise retrieval

Cons:
- little context
- more vectors
- more indexing overhead

Very large chunks:

Pros:
- more context
- fewer vectors

Cons:
- diluted relevance
- larger retrieval payloads

The right chunk size depends on:

document type
embedding model
query style
application requirements

There is no universal magic number.

A practical strategy is to start with moderate chunks and measure retrieval quality.

15. Overlap Can Preserve Context

Suppose we create chunks of 500 tokens.

We might overlap by 50 tokens:

Chunk 1
████████████████████████████████

                 overlap
                     █████
                         ███████████████████████████████
                         Chunk 2

The overlap helps prevent important information from being split exactly at a boundary.

Conceptually:

Document

A B C D E F G H I J K L M N

Chunk 1:
A B C D E F G

Chunk 2:
        F G H I J K L M

Chunk 3:
                L M N ...

The exact overlap should be measured rather than blindly chosen.

More overlap means:

more vectors
more storage
more embedding cost

But it may improve retrieval quality.

16. Building a Chunking Function

A simple educational implementation:

def chunk_text(text, chunk_size=1000, overlap=200):
    chunks = []

    start = 0

    while start < len(text):
        end = start + chunk_size

        chunk = text[start:end]

        chunks.append(chunk)

        start = end - overlap

    return chunks

This is intentionally simple.

A production system should preferably understand document structure rather than blindly slicing strings.

For example:

def structured_chunks(document):
    sections = split_by_heading(document)

    chunks = []

    for section in sections:
        chunks.extend(
            split_by_semantic_boundary(section)
        )

    return chunks

The important idea is:

Chunking is part of search quality, not just preprocessing.

17. The Embedding Pipeline

Now we can build the ingestion pipeline.

              Raw Document
                    │
                    ▼
              Clean Content
                    │
                    ▼
                Chunking
                    │
                    ▼
             Text Chunks
                    │
                    ▼
             Embedding Model
                    │
                    ▼
              Vector Embeddings
                    │
                    ▼
                PostgreSQL
                    │
                    ▼
              Vector Index

A document ingestion function might look like:

def index_document(document):
    chunks = chunk_text(
        document.content
    )

    for position, chunk in enumerate(chunks):

        vector = embedding_model.embed(
            chunk
        )

        save_chunk(
            document_id=document.id,
            position=position,
            content=chunk,
            embedding=vector
        )

This is conceptually simple.

But production ingestion needs to deal with:

retries
rate limits
partial failures
duplicate documents
model versioning
deleted documents
embedding migrations
background processing

18. Separate Documents From Chunks

A better schema is often:

CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    source TEXT,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Then:

CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,

    document_id BIGINT NOT NULL
        REFERENCES documents(id)
        ON DELETE CASCADE,

    chunk_index INTEGER NOT NULL,

    content TEXT NOT NULL,

    embedding VECTOR(1536),

    metadata JSONB NOT NULL DEFAULT '{}',

    created_at TIMESTAMPTZ
        NOT NULL DEFAULT now(),

    UNIQUE(document_id, chunk_index)
);

This gives us:

documents
    │
    ├── chunk 0
    ├── chunk 1
    ├── chunk 2
    └── chunk 3

Now search can return chunks while the application reconstructs the parent document.

19. Store Embedding Model Metadata

This is an underrated production concern.

Suppose you start with:

embedding-model-v1

Then six months later you migrate to:

embedding-model-v2

Your vector space may change.

You should know which model produced each vector.

For example:

ALTER TABLE document_chunks
ADD COLUMN embedding_model TEXT;

Then:

embedding_model = "model-v1"

This makes migrations much easier.

You may eventually need:

model-v1 vectors
model-v2 vectors

to coexist during migration.

20. Running a Semantic Query

Once vectors are stored, the search process becomes:

User Query
    │
    ▼
Embedding Model
    │
    ▼
Query Vector
    │
    ▼
Vector Search
    │
    ▼
Candidate Chunks
    │
    ▼
Metadata Filters
    │
    ▼
Reranking
    │
    ▼
Final Results

The query vector is generated once.

Then PostgreSQL compares it against stored embeddings.

With pgvector, cosine distance can be expressed using the cosine-distance operator.

Conceptually:

SELECT
    id,
    document_id,
    content,
    embedding <=> :query_embedding AS distance
FROM document_chunks
ORDER BY embedding <=> :query_embedding
LIMIT 10;

Smaller distance means greater similarity.

You can convert distance into a similarity-like score:

1 - (embedding <=> :query_embedding)

depending on the metric and interpretation you are using.

21. Add an Index

For larger collections, you don't want to perform a full scan forever.

pgvector supports approximate nearest-neighbor indexing approaches such as HNSW and IVFFlat.

For example, an HNSW index can be created conceptually as:

CREATE INDEX document_chunks_embedding_idx
ON document_chunks
USING hnsw (embedding vector_cosine_ops);

Then:

Query Vector
     │
     ▼
HNSW Index
     │
     ▼
Nearest candidates
     │
     ▼
Top K

The exact index choice and parameters should be benchmarked against your dataset.

A search engine is not finished when the query returns results.

It is finished when it returns good results quickly enough.

22. What Is HNSW?

HNSW stands for:

Hierarchical Navigable Small World.

It creates a graph structure that allows approximate nearest-neighbor navigation.

A simplified visualization:

Layer 2:

A ─────────────── D
 \                /
  \              /
   ─────── G ───


Layer 1:

A ── B ── C ── D ── E ── F ── G
 \    \       /     \       /
  ─────── H ───────────────

Instead of checking every vector, the search traverses the graph toward increasingly promising candidates.

Conceptually:

Query
 │
 ▼
Start node
 │
 ▼
Nearest neighbor
 │
 ▼
Better neighbor
 │
 ▼
Better neighbor
 │
 ▼
Candidate set

This can dramatically reduce search work.

But HNSW introduces tradeoffs involving:

memory
index build time
insertion cost
search quality
search latency

Again:

Benchmark instead of guessing.

23. Exact Search Is Still Valuable

Do not immediately assume approximate search is necessary.

If you have:

10,000 vectors

a sequential scan may be completely reasonable.

The architecture can be:

10k vectors
     │
     ▼
exact cosine search
     │
     ▼
fast enough

Introducing a complicated ANN index might add complexity without meaningful benefit.

This is a general engineering principle:

Optimize after measuring.

Semantic search is no exception.

24. Metadata Filtering Changes Everything

Suppose your database contains:

10 million documents

but the user should only search:

tenant_id = 42

You should not retrieve arbitrary documents and filter them in application code.

Instead:

SELECT
    id,
    document_id,
    content
FROM document_chunks
WHERE metadata->>'tenant_id' = '42'
ORDER BY embedding <=> :query_embedding
LIMIT 10;

This is especially important in multi-tenant systems.

The search pipeline becomes:

User Query
    │
    ▼
Query Embedding
    │
    ▼
Tenant Filter
    │
    ▼
Vector Search
    │
    ▼
Top K

Security and relevance intersect here.

A semantically perfect result from the wrong tenant is still a security vulnerability.

25. Treat Authorization as Part of Retrieval

Imagine:

Company A
 ├── public documents
 └── confidential documents

Company B
 ├── public documents
 └── confidential documents

The search engine must never return Company A's private document to Company B.

Do not do:

search globally
    │
    ▼
filter permissions later

if that can expose sensitive data through:

result counts
scores
snippets
logs
caching
timing

Instead, authorization constraints should participate in candidate retrieval.

Conceptually:

Query
 │
 ├── tenant
 ├── user
 ├── roles
 ├── permissions
 └── semantic vector
        │
        ▼
Authorized candidate set
        │
        ▼
Semantic ranking

This is one of the most important production lessons.

26. Semantic Search Alone Is Not Enough

Imagine the user searches:

"PostgreSQL"

They probably expect documents containing:

PostgreSQL

to rank highly.

But semantic search might retrieve:

relational database optimization

before:

PostgreSQL 18 release notes

because the semantic representation considers broader meaning.

This is why production systems often use hybrid search.

Combine:

lexical search
+
semantic search

27. Hybrid Search Architecture

A hybrid system might look like:

                    Query
                      │
             ┌────────┴────────┐
             │                 │
             ▼                 ▼
        Keyword Search    Semantic Search
             │                 │
             ▼                 ▼
         BM25 results      Vector results
             │                 │
             └────────┬────────┘
                      ▼
                Result Fusion
                      │
                      ▼
                   Reranker
                      │
                      ▼
                 Final Results

Keyword search is good at:

exact names
IDs
product codes
technical terms
rare words

Semantic search is good at:

paraphrases
conceptual similarity
natural-language questions
related concepts

Together they are much stronger.

28. BM25 + Vector Search

A common lexical ranking algorithm is BM25.

Conceptually:

BM25 score
+
semantic similarity

We can normalize the two scores and combine them:

[
Score =
\alpha \cdot SemanticScore
+
(1-\alpha)\cdot KeywordScore
]

For example:

[
Score =
0.7S_{semantic}
+
0.3S_{keyword}
]

This is only a starting point.

The correct weighting should be determined empirically.

29. Reciprocal Rank Fusion

Another approach is Reciprocal Rank Fusion.

Suppose:

Semantic ranking:

A
B
C
D


Keyword ranking:

C
A
D
B

RRF assigns a score based on rank.

Conceptually:

[

RRF(d)

\sum_i
\frac{1}{k + rank_i(d)}
]

This is useful because you don't necessarily need the scores from the two systems to be directly comparable.

You combine rankings instead.

Semantic results
       │
       ▼
     ranks
       │
       ├──────────┐
       │          │
Keyword results   │
       │          │
       ▼          │
     ranks        │
       │          │
       └────┬─────┘
            ▼
          RRF
            │
            ▼
       unified ranking

30. Reranking

Initial retrieval might return:

top 50 candidates

Then a more expensive model reranks them.

Why?

Vector similarity is useful, but it is still a relatively coarse measure.

A reranker can examine:

query
+
candidate document

and produce a more precise relevance score.

The architecture becomes:

Query
 │
 ▼
Embedding
 │
 ▼
Retrieve 100 candidates
 │
 ▼
Cheap ranking
 │
 ▼
Reranker
 │
 ▼
Top 10

This is a classic retrieval architecture.

You don't want to run an expensive reranker against millions of documents.

You use cheap retrieval first.

Then expensive reasoning on a small candidate set.

31. Search Is a Funnel

A useful mental model is:

                1,000,000 documents
                        │
                        ▼
                 Metadata filters
                        │
                        ▼
                  100,000 candidates
                        │
                        ▼
                 Vector retrieval
                        │
                        ▼
                    100 candidates
                        │
                        ▼
                  Reranking model
                        │
                        ▼
                     10 results
                        │
                        ▼
                   User interface

Every stage should reduce the search space.

This is how you make intelligent search affordable.

32. Building the Backend API

Let's build a simplified FastAPI endpoint.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class SearchRequest(BaseModel):
    query: str
    limit: int = 10


@app.post("/search")
def search(request: SearchRequest):

    query_vector = embedding_model.embed(
        request.query
    )

    results = search_database(
        query_vector,
        request.limit
    )

    return {
        "query": request.query,
        "results": results
    }

The backend flow is:

POST /search
     │
     ▼
validate request
     │
     ▼
generate embedding
     │
     ▼
query PostgreSQL
     │
     ▼
retrieve candidates
     │
     ▼
return results

But this simple endpoint hides several important engineering problems.

33. Embedding Generation Is Often the Expensive Part

Suppose your search endpoint receives:

100 requests/second

and every request requires an embedding API call.

Now you have:

100 embedding requests/sec

That introduces:

latency
cost
rate limits
external dependency

You can mitigate this with caching.

For example:

query
 │
 ▼
normalize
 │
 ▼
hash(query)
 │
 ▼
cache lookup
 │
 ├── hit ─────► vector
 │
 └── miss ────► embedding model
                     │
                     ▼
                   cache

A simple cache key might be:

semantic-search:v1:<hash>

The model version belongs in the cache key.

Otherwise:

model-v1 query vector

might accidentally be reused after migrating to:

model-v2

34. Query Normalization

Before embedding, you can normalize certain query forms.

For example:

"How do I build an API?"

and:

"how do i build an api?"

may be equivalent.

But be careful.

Aggressive normalization can destroy meaning.

For example:

"C++"

must not become:

"c"

And:

"Node.js"

should not become:

"node"

Search normalization should be driven by evaluation.

Don't normalize because it "looks cleaner."

Normalize because it improves retrieval.

35. Semantic Search Over Technical Documentation

Let's consider a real backend use case.

Suppose we have documentation:

PostgreSQL Tricks
Docker Deployment Guide
Django Authentication
React State Management
Kubernetes Networking

The user asks:

"How do I stop duplicate records from being created?"

The PostgreSQL article may contain:

UNIQUE constraints
ON CONFLICT
idempotency

There may be no exact phrase:

"stop duplicate records"

But semantic retrieval can connect:

duplicate records
        │
        ▼
idempotency
        │
        ▼
ON CONFLICT
        │
        ▼
PostgreSQL chunk

This is where semantic search becomes much more useful than a basic SQL LIKE.

36. Semantic Search Is Not Magic

Suppose the user searches:

"error code 0x80131500"

Semantic search might not be the best tool.

The exact identifier matters.

Similarly:

"invoice 847291"

requires lexical precision.

Or:

"API-KEY-ABC123"

is not a semantic concept.

This is why hybrid search is so powerful.

Exact identifiers
       │
       ▼
Keyword search


Natural language concepts
       │
       ▼
Semantic search

The right search engine uses the right retrieval mechanism for the query.

37. Build a Search Class

A cleaner architecture:

class SemanticSearch:

    def __init__(
        self,
        embedder,
        repository,
        reranker=None
    ):
        self.embedder = embedder
        self.repository = repository
        self.reranker = reranker

    def search(
        self,
        query,
        limit=10
    ):
        vector = self.embedder.embed(
            query
        )

        candidates = self.repository.vector_search(
            vector,
            limit=50
        )

        if self.reranker:
            candidates = self.reranker.rank(
                query,
                candidates
            )

        return candidates[:limit]

This separates:

embedding
retrieval
reranking

That separation becomes very useful as the system grows.

38. Repository Layer

Your database code should not be scattered through API controllers.

For example:

class DocumentRepository:

    def vector_search(
        self,
        vector,
        limit=10
    ):
        sql = """
            SELECT
                id,
                document_id,
                content,
                1 - (embedding <=> %s)
                    AS similarity
            FROM document_chunks
            ORDER BY embedding <=> %s
            LIMIT %s
        """

        return execute(
            sql,
            [vector, vector, limit]
        )

Now your application architecture becomes:

FastAPI
   │
   ▼
Search Service
   │
   ├── Embedding Provider
   │
   ├── Repository
   │
   └── Reranker
          │
          ▼
      PostgreSQL

This keeps infrastructure replaceable.

39. Background Indexing

Don't generate embeddings synchronously during every document upload if the process is expensive.

Instead:

POST /documents
       │
       ▼
store document
       │
       ▼
create indexing job
       │
       ▼
return 202
       │
       ▼
background worker
       │
       ├── chunk
       ├── embed
       └── store vectors

This makes the user-facing API faster.

A PostgreSQL-backed job queue can even work for smaller systems.

documents
    │
    ▼
embedding_jobs
    │
    ▼
worker
    │
    ▼
document_chunks

The document becomes searchable once indexing completes.

40. Idempotent Indexing

Suppose the worker crashes halfway through indexing.

You don't want:

document 42
 ├── chunk 1
 ├── chunk 2
 ├── chunk 3
 ├── chunk 3
 ├── chunk 4
 └── chunk 4

Use a unique constraint:

UNIQUE(document_id, chunk_index)

Then indexing can safely use:

INSERT INTO document_chunks (...)
VALUES (...)
ON CONFLICT (
    document_id,
    chunk_index
)
DO UPDATE SET
    content = EXCLUDED.content,
    embedding = EXCLUDED.embedding;

Now retries become safe.

This is the same backend engineering principle we use everywhere:

Make expensive operations idempotent.

41. Versioning the Index

Suppose your chunking strategy changes.

Version 1:

chunk_size = 500

Version 2:

chunk_size = 800

Your embedding model changes.

Your metadata extraction changes.

Your preprocessing changes.

You need to know which pipeline generated which vector.

Add:

embedding_version TEXT NOT NULL

and perhaps:

chunking_version TEXT NOT NULL

Now you can reason about your search corpus.

Document
   │
   ├── pipeline v1
   │      └── vectors
   │
   └── pipeline v2
          └── vectors

This is essential for controlled migrations.

42. Evaluation Is More Important Than the Demo

A semantic search demo can look incredible.

You enter:

"How do I secure my API?"

and get:

API Security

Perfect.

But production search has thousands of queries.

You need evaluation.

Create a dataset:

query                         relevant_docs

"secure API endpoints"        [12, 44]
"postgres duplicate records" [8]
"docker deployment"           [19, 21]
"database transaction"        [4, 7]

Then measure retrieval quality.

43. Precision and Recall

Precision asks:

Of the documents returned, how many are relevant?

[
Precision =
\frac{Relevant\ Retrieved}
{Retrieved}
]

Recall asks:

Of all relevant documents, how many did we retrieve?

[
Recall =
\frac{Relevant\ Retrieved}
{Relevant}
]

For example:

Retrieved = 10
Relevant retrieved = 8

Then:

[
Precision = 0.8
]

If there were 20 relevant documents overall:

[
Recall = 8/20 = 0.4
]

This tells you something important.

Your system can be precise but incomplete.

Or broad but noisy.

44. Top-K Metrics

Search systems often care about:

Precision@K
Recall@K
MRR
NDCG

For example:

Precision@5

asks:

How many of the first five results are relevant?

This is often more meaningful than overall precision because users rarely inspect result 200.

A search system should optimize for the top of the ranking.

45. Human Evaluation Still Matters

Metrics are useful.

But search quality is ultimately about users.

Give real users queries and ask:

Was result #1 useful?
Was result #2 useful?
Did the system answer your intent?
Did you find what you wanted?

You might discover:

Vector similarity = excellent
User satisfaction = poor

Why?

Maybe the results are semantically similar but not actionable.

This is why semantic search should be evaluated as a product feature, not merely an ML benchmark.

46. Common Failure: Bad Chunking

Suppose a document says:

PostgreSQL uses MVCC.

This means transactions can operate concurrently
without readers blocking writers in many cases.

If chunking produces:

Chunk 1:
PostgreSQL uses MVCC.

Chunk 2:
This means transactions...

the second chunk may lose important context.

Better:

Chunk 1:
PostgreSQL uses MVCC. This means transactions
can operate concurrently without readers blocking
writers in many cases.

The embedding now captures the concept more completely.

Search quality often improves more from better chunking than from blindly switching models.

47. Common Failure: Embedding the Wrong Content

Suppose a product database contains:

name
price
stock
description
category

You generate embeddings from:

"19.99"

That is useless.

Instead:

Product:
Mechanical Keyboard

Category:
Computer Accessories

Description:
Compact mechanical keyboard designed for
software developers and gamers...

Then embed the semantic content.

You can separately store structured fields for filtering.

Embedding:
meaning


Metadata:
price
category
availability
tenant
permissions

This separation is powerful.

48. Structured Data and Semantic Data Should Coexist

Imagine an e-commerce search:

"black laptop for programming under $1500"

The semantic part is:

laptop
programming

The structured constraints are:

color = black
price <= 1500

A good search architecture does:

Natural language query
        │
        ├───────────────┐
        │               │
        ▼               ▼
semantic concepts   structured filters
        │               │
        └───────┬───────┘
                ▼
         candidate search
                │
                ▼
             ranking

This is one of the most important patterns for production search.

49. Query Understanding

You can explicitly extract structured constraints.

For example:

"black laptop for programming under $1500"

becomes:

{
  "semantic_query": "laptop for programming",
  "filters": {
    "color": "black",
    "price_max": 1500
  }
}

Then:

semantic_query
      │
      ▼
embedding
      │
      ▼
vector search

filters
      │
      ▼
SQL predicates

This produces much better results than trying to make one vector represent everything.

50. Semantic Search and RAG

Semantic search is also one of the foundations of Retrieval-Augmented Generation.

The architecture looks like:

User Question
      │
      ▼
Query Embedding
      │
      ▼
Semantic Retrieval
      │
      ▼
Relevant Chunks
      │
      ▼
Context Window
      │
      ▼
Language Model
      │
      ▼
Generated Answer

This is where semantic search becomes particularly powerful.

Instead of asking a language model to know everything, we retrieve relevant knowledge from our own corpus.

For example:

Question:
"What is our company's refund policy?"

Semantic search retrieves:

Refund Policy
Section 4

Then the model uses that content to formulate the answer.

51. But RAG Quality Is Search Quality

A common misconception is:

"If the language model is powerful enough, it will fix bad retrieval."

Usually it won't.

If retrieval gives the model:

wrong documents

the model has limited ability to magically discover the correct source.

Therefore:

RAG quality
    ≈
retrieval quality
+
generation quality

If retrieval is broken, generation can become confidently wrong.

This is why search engineering is becoming increasingly important in AI systems.

52. Security in Semantic Search

Vector databases contain data representations derived from your documents.

Those vectors are not automatically harmless.

If your corpus contains:

private customer data
internal documents
financial information
API documentation
credentials

your vector index becomes part of your sensitive data infrastructure.

Security must include:

tenant isolation
authentication
authorization
encryption at rest
encryption in transit
audit logging
access controls
deletion workflows
retention policies

And don't forget the source documents.

Deleting a document should also remove:

document
chunks
embeddings
cache entries
search indexes
derived artifacts

Data deletion becomes a pipeline.

53. The Deletion Problem

Suppose a customer asks:

"Delete all my data."

You remove the document from the primary table.

But perhaps you still have:

document_chunks
embedding cache
search cache
analytics logs
backup
vector index

The search system can accidentally continue returning deleted information.

Therefore, deletion must propagate.

Delete document
      │
      ├── delete chunks
      │
      ├── invalidate caches
      │
      ├── remove search references
      │
      └── schedule backup lifecycle

This is another reason to keep the architecture understandable.

54. Caching Search Results

Caching can exist at several layers.

Query embedding cache

query
 │
 ▼
embedding cache
 │
 ▼
vector

Search result cache

query + filters + model_version
           │
           ▼
        cache
           │
           ▼
        results

Document cache

chunk ID
   │
   ▼
content cache

But cache keys must include all relevant parameters.

For example:

tenant_id
query
filters
model_version
index_version
top_k

Otherwise, one user's results might accidentally be returned to another user.

That's not a performance bug.

That's a security incident.

55. Observability

A production semantic search system should measure:

embedding latency
database latency
reranking latency
total latency
cache hit rate
result count
score distribution
zero-result rate
click-through rate

A useful trace might look like:

/search
 │
 ├── embedding: 72ms
 │
 ├── PostgreSQL: 18ms
 │
 ├── reranker: 110ms
 │
 └── total: 205ms

Now you know where the bottleneck is.

Without observability, you are guessing.

56. Score Distributions Matter

Suppose every query produces:

Result 1: 0.91
Result 2: 0.90
Result 3: 0.89

That may indicate strong semantic matches.

But another query might produce:

Result 1: 0.42
Result 2: 0.41
Result 3: 0.40

Should you still return them?

Maybe.

Maybe not.

A fixed:

top_k = 10

doesn't guarantee relevance.

Sometimes you need a threshold:

if similarity < 0.55:
    don't return

But the correct threshold depends on:

embedding model
corpus
query distribution
metric
normalization

Again, evaluate empirically.

57. Semantic Search Is a Ranking System

At its core, the search engine is producing:

(query, document) → score

Then:

sort(document, score)

But production systems often have multiple signals:

semantic similarity
keyword relevance
freshness
authority
popularity
permissions
business rules
user preferences

A more realistic ranking function might be:

[
Score =
w_1 Semantic
+
w_2 Keyword
+
w_3 Freshness
+
w_4 Authority
+
w_5 Popularity
]

The weights are learned or tuned from evaluation data.

This is where search starts becoming information retrieval engineering rather than simply vector similarity.

58. Freshness Can Matter

Imagine searching:

"latest PostgreSQL security update"

An old article may be semantically perfect.

But it is not useful if the user needs current information.

Therefore:

semantic relevance
+
recency

can be more useful.

For example:

[
FinalScore =
0.8 \cdot SemanticScore
+
0.2 \cdot FreshnessScore
]

The correct weighting depends on the application.

For legal, financial, security, and technical content, freshness can be especially important.

59. Popularity Can Be Dangerous

Suppose your system always boosts popular documents.

Then:

popular

can overpower:

relevant

This creates a feedback loop:

popular document
      │
      ▼
higher ranking
      │
      ▼
more clicks
      │
      ▼
more popularity
      │
      └───────────────┐
                      ▼
                 even higher rank

Search ranking should be carefully designed to avoid reinforcing irrelevant content.

60. Build a Production Architecture

Let's put everything together.

                         ┌──────────────────┐
                         │     Client       │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │      API         │
                         │    FastAPI       │
                         └────────┬─────────┘
                                  │
                    ┌─────────────┴────────────┐
                    │                          │
                    ▼                          ▼
              Query Service              Auth Service
                    │
                    ▼
              Query Embedding
                    │
                    ▼
            ┌──────────────────┐
            │ Retrieval Layer  │
            └────────┬─────────┘
                     │
          ┌──────────┴──────────┐
          │                     │
          ▼                     ▼
     PostgreSQL            Lexical Search
     + pgvector                 │
          │                     │
          └──────────┬──────────┘
                     ▼
               Candidate Merge
                     │
                     ▼
                 Reranker
                     │
                     ▼
                Final Ranking
                     │
                     ▼
                   Cache
                     │
                     ▼
                  Response

The ingestion pipeline is separate:

Documents
    │
    ▼
Ingestion API
    │
    ▼
Job Queue
    │
    ▼
Workers
    │
    ├── parse
    ├── clean
    ├── chunk
    ├── embed
    └── persist
            │
            ▼
        PostgreSQL

This separation is important.

Search should not be blocked by ingestion.

61. A More Complete PostgreSQL Schema

Here is a practical starting point:

CREATE EXTENSION IF NOT EXISTS vector;


CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,

    tenant_id BIGINT NOT NULL,

    title TEXT NOT NULL,

    source TEXT,

    metadata JSONB NOT NULL DEFAULT '{}',

    content_hash TEXT NOT NULL,

    created_at TIMESTAMPTZ
        NOT NULL DEFAULT now(),

    updated_at TIMESTAMPTZ
        NOT NULL DEFAULT now(),

    UNIQUE (
        tenant_id,
        content_hash
    )
);


CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,

    document_id BIGINT NOT NULL
        REFERENCES documents(id)
        ON DELETE CASCADE,

    chunk_index INTEGER NOT NULL,

    content TEXT NOT NULL,

    embedding VECTOR(1536),

    embedding_model TEXT NOT NULL,

    chunking_version TEXT NOT NULL,

    metadata JSONB NOT NULL DEFAULT '{}',

    created_at TIMESTAMPTZ
        NOT NULL DEFAULT now(),

    UNIQUE (
        document_id,
        chunk_index
    )
);

Then:

CREATE INDEX idx_document_chunks_document
ON document_chunks(document_id);

And:

CREATE INDEX idx_document_chunks_embedding
ON document_chunks
USING hnsw (embedding vector_cosine_ops);

The exact vector dimension and index configuration should match your selected embedding model and workload.

62. A Complete Search Query

Suppose we want tenant-specific search.

SELECT
    c.id,
    c.document_id,
    c.content,
    d.title,

    1 - (
        c.embedding <=> :query_embedding
    ) AS similarity

FROM document_chunks c

JOIN documents d
    ON d.id = c.document_id

WHERE d.tenant_id = :tenant_id

ORDER BY c.embedding <=> :query_embedding

LIMIT :limit;

This is already a useful semantic search backend.

But production systems often go further:

tenant filter
+
permission filter
+
semantic search
+
keyword search
+
reranking
+
freshness

The vector query is only one stage.

63. Search API Response Design

Don't just return:

[
  {
    "content": "..."
  }
]

Return useful metadata.

For example:

{
  "query": "how do I prevent duplicate records?",
  "results": [
    {
      "document_id": 42,
      "chunk_id": 801,
      "title": "PostgreSQL Tricks",
      "content": "Use ON CONFLICT...",
      "score": 0.91,
      "metadata": {
        "category": "database"
      }
    }
  ]
}

This allows the frontend to show:

PostgreSQL Tricks

Use ON CONFLICT to safely handle
duplicate insert attempts...

Similarity: 0.91

But be careful about exposing raw similarity scores to users.

They are often meaningful internally but not necessarily intuitive as user-facing confidence values.

64. Search Highlighting Still Matters

Even semantic search benefits from highlighting.

Suppose the result is:

PostgreSQL supports ON CONFLICT...

The UI could show:

PostgreSQL supports **ON CONFLICT**
for safely handling duplicate records...

This helps users understand why the result was returned.

Semantic ranking tells us:

"This is probably relevant."

Highlighting tells the user:

"Here is the part that might matter."

These are different jobs.

65. The Danger of Hallucinated Relevance

A semantic search engine can return a result that feels related but does not actually answer the query.

Example:

Query:

"How do I configure PostgreSQL replication?"

Result:

"PostgreSQL backup strategies"

Related?

Yes.

Answer?

Not necessarily.

This is why reranking and evaluation matter.

Semantic similarity is not the same thing as usefulness.

66. Build Search Around User Intent

A better system tries to identify intent.

For example:

"How do I configure replication?"

Intent:

technical how-to

Query:

"PostgreSQL replication"

Another:

"Why is my PostgreSQL query slow?"

Intent:

troubleshooting

The retrieval system might prioritize:

EXPLAIN ANALYZE
indexes
query plans
locks

This is where search becomes more intelligent than simply comparing embeddings.

67. Semantic Search and Backend Engineering

Semantic search is an interesting intersection of several disciplines.

                Semantic Search
                       │
        ┌──────────────┼───────────────┐
        │              │               │
        ▼              ▼               ▼
 Information      Machine          Backend
 Retrieval        Learning         Engineering
        │              │               │
        ▼              ▼               ▼
 ranking         embeddings        APIs
 indexing        models            databases
 precision       inference         caching
 recall          evaluation        concurrency

This is why building semantic search is such a useful backend project.

You learn machine learning without needing to train a foundation model.

You learn databases without abandoning application engineering.

And you learn search without having to build Google.

68. The Most Important Optimization: Reduce Work

Suppose you have:

100 million vectors

Your biggest optimization is not:

make cosine similarity 10% faster

It is:

avoid comparing against 99.99% of vectors

This leads to:

filter
→ retrieve
→ rerank

rather than:

compare everything
→ sort everything

Systems engineering is often about avoiding unnecessary work.

Semantic search makes this principle very visible.

69. When You Should Use a Dedicated Vector Database

PostgreSQL + pgvector is excellent for many applications.

But eventually you may need specialized infrastructure.

Consider a dedicated vector system when you have requirements around:

enormous vector collections
specialized ANN workloads
very high search throughput
distributed vector indexing
specialized filtering and retrieval features
independent scaling of vector search

The architecture can evolve:

Early stage

PostgreSQL
 ├── documents
 ├── metadata
 └── vectors


Later stage

PostgreSQL
 │
 ├── authoritative data
 │
 └── metadata

Vector system
 │
 └── embeddings

But introducing another database creates synchronization problems.

Therefore:

Don't introduce a vector database because semantic search is fashionable.

Introduce it because your workload requires it.

70. Database First, Search Second

This is a principle I like for backend architecture.

Your authoritative data should remain authoritative.

The embedding is a derived representation.

Source document
      │
      ▼
embedding pipeline
      │
      ▼
vector

If the vector disappears:

rebuild

You should not lose the underlying document.

This makes embeddings similar to:

cache
index
materialized view
derived state

That mental model is extremely useful.

71. Embeddings Are Derived State

Think of the relationship:

Document
   │
   ├── title
   ├── content
   └── metadata
          │
          ▼
      embedding model
          │
          ▼
       embedding

If you change:

embedding model

you should be able to regenerate the vectors.

If you change:

chunking strategy

you should be able to regenerate the chunks.

This means your pipeline should be reproducible.

That is a very important production requirement.

72. Reproducibility

Store:

document hash
embedding model
embedding version
chunking version
preprocessing version

Then you can answer:

"Why does this document have this vector?"

You should be able to trace:

document
   │
   ▼
content hash
   │
   ▼
preprocessing v3
   │
   ▼
chunking v2
   │
   ▼
embedding-model-v4
   │
   ▼
vector

This is search observability at the data level.

73. Semantic Search Is a Pipeline, Not a Feature

It is tempting to describe semantic search as:

vector database

That is incomplete.

A real system is:

                 ┌─────────────┐
                 │   Sources   │
                 └──────┬──────┘
                        │
                        ▼
                 ┌─────────────┐
                 │  Parsing    │
                 └──────┬──────┘
                        │
                        ▼
                 ┌─────────────┐
                 │  Chunking   │
                 └──────┬──────┘
                        │
                        ▼
                 ┌─────────────┐
                 │ Embeddings  │
                 └──────┬──────┘
                        │
                        ▼
                 ┌─────────────┐
                 │ Vector DB   │
                 └──────┬──────┘
                        │
                        ▼
                   Retrieval
                        │
                        ▼
                    Ranking
                        │
                        ▼
                    Reranking
                        │
                        ▼
                    Results

If any stage is poor, the user experience suffers.

74. A Practical Development Roadmap

If I were building semantic search from scratch, I would not begin with a huge architecture.

I would build it in stages.

Stage 1

PostgreSQL
+
pgvector
+
embedding model

Implement:

document ingestion
chunking
embedding
vector search

Stage 2

Add:

metadata filtering
caching
background workers

Stage 3

Add:

keyword search
hybrid retrieval

Stage 4

Add:

reranking

Stage 5

Build:

evaluation dataset
metrics
observability

Stage 6

Optimize:

ANN indexes
query latency
embedding costs
cache hit rates

Stage 7

Only then consider:

dedicated vector infrastructure

This progression keeps the architecture understandable.

75. Final Mental Model

If you remember only one diagram from this article, remember this:

                           USER
                            │
                            ▼
                         QUERY
                            │
                            ▼
                    Query Understanding
                            │
                 ┌──────────┴──────────┐
                 │                     │
                 ▼                     ▼
             Keywords             Embedding
                 │                     │
                 ▼                     ▼
          Lexical Search        Vector Search
                 │                     │
                 └──────────┬──────────┘
                            ▼
                       Candidates
                            │
                            ▼
                       Reranking
                            │
                            ▼
                     Business Rules
                            │
                            ▼
                     Final Results
                            │
                            ▼
                          USER

And the data pipeline:

                      DOCUMENT
                          │
                          ▼
                     CLEANING
                          │
                          ▼
                      CHUNKING
                          │
                          ▼
                     EMBEDDING
                          │
                          ▼
                  VECTOR + METADATA
                          │
                          ▼
                    VECTOR INDEX
                          │
                          ▼
                     RETRIEVAL

These two pipelines form the heart of semantic search.

Final Thoughts

Semantic search looks like an AI problem.

It is.

But it is also a database problem.

It is also an information retrieval problem.

It is also a ranking problem.

And, perhaps most importantly for backend developers, it is a systems engineering problem.

The first breakthrough is understanding that text can be represented as vectors.

The second is realizing that vectors create geometry.

The third is realizing that search can become nearest-neighbor retrieval.

Then the real engineering begins.

You need to decide how documents are chunked.

You need to decide how embeddings are generated.

You need to store them efficiently.

You need to retrieve them quickly.

You need to filter by permissions.

You need to combine semantic and lexical signals.

You need to rerank candidates.

You need to evaluate relevance.

You need to cache expensive operations.

You need to monitor latency.

You need to version your embedding pipeline.

You need to handle deletion.

You need to prevent tenant leakage.

And you need to know when semantic search is simply the wrong tool.

The most important lesson is this:

An embedding is not search.

An embedding is a representation.

The search engine is the system built around that representation.

That distinction changes how you architect the entire application.

A weak implementation says:

text
 ↓
embedding
 ↓
database
 ↓
results

A mature implementation looks more like:

                    Documents
                        │
                        ▼
                 Parsing / Cleaning
                        │
                        ▼
                     Chunking
                        │
                        ▼
                    Embeddings
                        │
                        ▼
              PostgreSQL + pgvector
                        │
                        ▼
                  Query Embedding
                        │
            ┌───────────┴───────────┐
            │                       │
            ▼                       ▼
       Lexical Search          Vector Search
            │                       │
            └───────────┬───────────┘
                        ▼
                   Candidate Set
                        │
                        ▼
                    Reranking
                        │
                        ▼
                Authorization Rules
                        │
                        ▼
                 Final Results

And this architecture reveals something deeper.

Search is fundamentally about reducing uncertainty.

The user has a question.

The system has millions of possible documents.

The job of the search engine is to progressively narrow that enormous space until a small number of highly relevant pieces of information remain.

Keyword search narrows the space through words.

Semantic search narrows it through meaning.

Hybrid search uses both.

Reranking applies deeper judgment.

Metadata filters apply hard constraints.

And the final ranking becomes the interface between an enormous information space and a human being.

That is why semantic search is such an interesting engineering problem.

You are not simply storing vectors.

You are building a system that tries to understand what a user means, map that meaning into mathematical space, navigate an enormous collection of representations, apply business constraints, rank competing interpretations, and return useful information fast enough that the user never notices the complexity underneath.

The code may eventually look surprisingly small.

The architecture behind it is not.

And that is the real lesson of building semantic search:

The difficult part isn't generating the vector.

The difficult part is turning a vector into a trustworthy search experience.