DEV Community: Derek mwale

The 5 Code Reviews That Changed How I Think About Engineering

Derek mwale — Fri, 07 Aug 2026 06:06:33 +0000

There are code reviews you forget five minutes after approving them.

Then there are code reviews that permanently change how you think about software.

I've had both.

Early in my career, I thought code review was mostly about catching bugs.

Someone writes code.

Someone else reads it.

They find something wrong.

They leave a comment.

The developer fixes it.

Pull request gets merged.

Done.

That mental model is incomplete.

The best code reviews I've experienced weren't really about syntax, formatting, or whether a function was ten lines too long.

They were about how to think.

A good review can expose an architectural assumption you didn't realize you were making.

It can reveal that your data model doesn't represent reality.

It can show you that an apparently harmless abstraction will become a nightmare six months later.

It can demonstrate why an API should be designed around behavior rather than implementation.

And sometimes, the most valuable review comment is only one sentence long.

Over time, I started realizing that experienced engineers don't necessarily write better code because they know more syntax.

They write better code because they have developed better instincts about:

boundaries
failure
data
complexity
ownership
change
observability
trade-offs

These instincts are difficult to learn from tutorials.

You usually learn them by getting burned.

This article is about five hypothetical-but-realistic code reviews based on the kinds of engineering lessons that changed how I approach software.

Each review taught me something different.

And together they changed my definition of what "good code" actually means.

1. The Code Review That Taught Me That Correctness Comes Before Cleverness

The first lesson was simple:

Code that looks intelligent is not necessarily code that is correct.

I remember working on logic that involved processing a collection of records.

The implementation was compact.

It used a clever combination of filtering, mapping and reduction.

Something like:

total = sum(
    item.price * item.quantity
    for item in items
    if item.active
)

At first glance, this looks great.

It's short.

It's readable.

It's idiomatic Python.

But the review wasn't about the syntax.

The reviewer asked:

"Why are inactive products excluded from the calculation?"

That question stopped me.

I had assumed:

inactive product
      ↓
don't include it

But the business requirement was actually:

Product becomes inactive
      ↓
New purchases cannot use it
      ↓
Existing orders must preserve it

The calculation wasn't supposed to ask whether the product was currently active.

It was supposed to use the historical state captured when the order was created.

The bug wasn't in the loop.

The bug was in my mental model.

Code Can Be Locally Correct and Globally Wrong

This is one of the most important engineering lessons I've learned.

Consider:

def calculate_total(items):
    return sum(
        item.price * item.quantity
        for item in items
    )

The function itself may be perfectly correct.

But what if:

item.price

is mutable?

Suppose:

Monday:
Laptop = $1000

Tuesday:
Laptop = $1200

A customer bought it Monday.

If we calculate their historical order using the current product price:

Order total = $1200

we have created a logical bug.

The arithmetic is correct.

The system is wrong.

This is the difference between local correctness and system correctness.

The Fix Was a Data Modeling Decision

Instead of relying on the current product:

OrderItem
    │
    └── Product
           │
           └── current price

we capture the price at purchase time:

Order
 │
 └── OrderItem
       ├── product_id
       ├── quantity
       └── price_at_purchase

For example:

CREATE TABLE order_items (
    id BIGSERIAL PRIMARY KEY,

    order_id BIGINT NOT NULL
        REFERENCES orders(id),

    product_id BIGINT NOT NULL
        REFERENCES products(id),

    quantity INTEGER NOT NULL
        CHECK (quantity > 0),

    price_cents INTEGER NOT NULL
        CHECK (price_cents >= 0)
);

Now:

Product.price

represents the current catalog price.

While:

OrderItem.price_cents

represents historical truth.

That tiny modeling decision has enormous consequences.

The Diagram

             PRODUCT
        ┌────────────────┐
        │ id             │
        │ name           │
        │ current_price  │
        └───────┬────────┘
                │
                │ reference
                ▼
          ORDER_ITEM
        ┌────────────────┐
        │ product_id     │
        │ quantity       │
        │ price_cents    │
        └────────────────┘
                │
                ▼
             ORDER

The review taught me something bigger than "store historical prices."

It taught me:

Before optimizing code, make sure the code represents the correct model of reality.

The Engineering Lesson

A lot of bad software starts with developers asking:

"How do I implement this?"

before asking:

"What is actually true?"

That's backwards.

I now try to ask:

What does this data mean?
        ↓
What must always be true?
        ↓
What changes over time?
        ↓
What is historical?
        ↓
What is current?
        ↓
Then:
How should I implement it?

This is one reason I became much more interested in data modeling and system design.

The code is downstream from the model.

If the model is wrong, elegant code simply produces the wrong answer faster.

Implementation Principle: Make Invariants Explicit

Suppose an order quantity must always be positive.

Don't rely only on:

if quantity <= 0:
    raise ValueError()

You can also enforce it at the database level:

CHECK (quantity > 0)

Now you have:

Application validation
        +
Database constraint
        =
Defense in depth

The system becomes harder to corrupt.

That review changed how I see constraints.

Constraints aren't obstacles.

They're executable statements about reality.

2. The Code Review That Taught Me That Abstractions Have a Cost

The second review taught me something even more painful:

Not every abstraction makes software better.

I once encountered the temptation to create a generic repository layer.

It looked sophisticated.

Something like:

class BaseRepository:

    def find(self, filters):
        ...

    def create(self, data):
        ...

    def update(self, id, data):
        ...

    def delete(self, id):
        ...

Then:

class UserRepository(BaseRepository):
    pass

class ProductRepository(BaseRepository):
    pass

class OrderRepository(BaseRepository):
    pass

At first:

Less duplication!

Beautiful.

Then requirements changed.

Orders needed:

transactional operations
state transitions
locking
historical records

Users needed:

email uniqueness
authentication
soft deletion

Products needed:

inventory
pricing
availability

The generic abstraction started becoming:

BaseRepository
       │
       ├── if User
       ├── if Product
       ├── if Order
       ├── if special case
       └── if another special case

Eventually the abstraction was more complicated than the duplication it was supposed to eliminate.

The Review Comment

The reviewer essentially asked:

"Are these things actually the same, or do they just look similar right now?"

That changed how I think about abstraction.

Two pieces of code being structurally similar does not mean they share the same concept.

This distinction is enormous.

Similarity Is Not Identity

Imagine:

user.update(...)
product.update(...)
order.update(...)

They all technically update something.

But semantically:

User update
     ≠
Product update
     ≠
Order update

The abstraction:

update(entity)

may hide meaningful differences.

A better design might be:

user_service.update_profile(...)
product_service.change_price(...)
order_service.cancel(...)

Now the operations describe domain behavior.

The Diagram

Bad abstraction:

             BaseRepository
                  │
        ┌─────────┼─────────┐
        ▼         ▼         ▼
      User     Product     Order
        │         │         │
        └───── shared ──────┘

Eventually:

             BaseRepository
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
     User       Product     Order
       │          │           │
    hacks       hacks       hacks
       │          │           │
       └──────────┼───────────┘
                  ▼
            abstraction hell

Better:

UserService
ProductService
OrderService
      │
      ▼
Shared infrastructure
      │
      ▼
Database

Share infrastructure where it is genuinely shared.

Don't force domain concepts into the same shape.

Abstraction Should Follow Understanding

This became one of my favorite engineering principles.

Don't abstract what you haven't understood.

Early in a project, duplication can be useful.

It gives you information.

Suppose you have:

send_user_email()
send_order_email()
send_payment_email()

They look similar.

You could immediately create:

send_email(template, recipient, context)

Maybe that's correct.

But perhaps six months later you discover:

User email
→ transactional

Order email
→ retryable

Payment email
→ audit-sensitive

Now the generic abstraction becomes restrictive.

Sometimes three similar implementations are telling you:

"We don't yet know whether these things are actually the same."

That's valuable information.

The Rule I Use Now

I don't ask:

"Can I remove duplication?"

I ask:

"Does the duplication represent the same concept?"

If yes:

Abstract.

If no:

Keep them separate.

If uncertain:

Wait.

Waiting is sometimes the better engineering decision.

3. The Code Review That Taught Me to Design for Failure

The third review changed my relationship with error handling.

I had written code around an external API.

The happy path looked something like:

response = payment_provider.charge(
    amount=amount
)

if response.success:
    mark_order_paid(order)

Simple.

The reviewer asked:

"What happens if the payment succeeds but this process crashes before mark_order_paid()?"

That question opened an entire world.

Distributed Systems Exist Even in Small Applications

The system looked like:

Your API
   │
   ▼
Payment Provider
   │
   ▼
Payment

But now there are two independent systems.

Your database might say:

PENDING

while the payment provider says:

PAID

The network can fail.

The process can crash.

The response can disappear.

The user can retry.

The payment provider can timeout.

You don't control any of these.

The Naive Implementation

def pay(order):

    result = payment.charge(
        amount=order.total
    )

    if result.success:
        order.status = "PAID"
        db.save(order)

Potential sequence:

1. Request payment
2. Provider charges customer
3. Provider returns success
4. Application crashes
5. Database still says PENDING

Now what?

The Better Design

Introduce explicit payment states:

PENDING
AUTHORIZED
PAID
FAILED
REFUNDED

And idempotency:

Client
  │
  │ Idempotency-Key: abc123
  ▼
Backend
  │
  ▼
Payment Provider

Store:

abc123 → payment result

If the request arrives again:

abc123
  │
  ▼
Already processed
  │
  ▼
Return existing result

Implementation

A simplified schema:

CREATE TABLE payments (
    id BIGSERIAL PRIMARY KEY,

    order_id BIGINT NOT NULL
        REFERENCES orders(id),

    idempotency_key TEXT NOT NULL UNIQUE,

    status TEXT NOT NULL,

    amount_cents INTEGER NOT NULL
        CHECK (amount_cents > 0),

    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Then:

def create_payment(order, key):

    existing = payment_repo.find_by_key(key)

    if existing:
        return existing

    payment = payment_repo.create(
        order_id=order.id,
        idempotency_key=key,
        status="PENDING",
        amount_cents=order.total
    )

    return payment

The unique constraint protects against duplicates.

The Deeper Lesson

The review wasn't really about payments.

It taught me:

Assume the world can interrupt your program at any moment.

A process can die.

A machine can restart.

A database can timeout.

A network packet can disappear.

A user can click twice.

A worker can execute the same job twice.

An external API can return success after you timeout.

Once you think this way, software design changes.

You stop asking:

"Does this work?"

and start asking:

"What happens when this works halfway?"

That is a much more important question.

Design Around Failure States

A robust system doesn't pretend failure doesn't happen.

It models failure.

             Operation
                 │
        ┌────────┼────────┐
        ▼        ▼        ▼
     Success   Timeout   Failure
        │        │        │
        ▼        ▼        ▼
      Commit    Retry    Recover

And sometimes:

Unknown outcome
      │
      ▼
Reconcile
      │
      ▼
Determine truth

This is especially important for:

Payments
Orders
Messaging
Distributed jobs
File uploads
External APIs
Inventory

4. The Code Review That Taught Me That Complexity Is a Budget

The fourth review was about performance.

I had optimized something before it needed optimization.

The code was relatively straightforward.

But I thought:

"This will eventually be slow."

So I introduced caching.

API
 │
 ▼
Redis
 │
 ▼
PostgreSQL

The reviewer asked:

"What evidence tells us the database is the bottleneck?"

I didn't have any.

I had optimized a problem that didn't exist.

Premature Optimization Is Really Premature Complexity

People often quote:

"Premature optimization is the root of all evil."

I think the more practical version is:

Premature complexity is expensive even when the optimization is technically correct.

Caching introduces:

Cache keys
TTL
Invalidation
Stale data
Memory usage
Failure modes
Monitoring

Now instead of:

Request
 ↓
Database

you have:

Request
 ↓
Cache
 ├── HIT
 │    ↓
 │   Response
 │
 └── MISS
      ↓
    Database
      ↓
    Cache
      ↓
   Response

That's not free.

Measure First

Suppose we have:

SELECT *
FROM products
WHERE category_id = 42;

Before caching, measure.

Maybe the query takes:

4ms

Maybe there are:

2,000 requests/day

Caching this might provide almost no value.

Instead, suppose the query takes:

1.8 seconds

and receives:

50,000 requests/minute

Now we have evidence.

Optimization becomes engineering instead of speculation.

A Better Optimization Loop

Observe
  ↓
Measure
  ↓
Identify bottleneck
  ↓
Hypothesize
  ↓
Change
  ↓
Measure
  ↓
Keep or revert

This is very different from:

Guess
 ↓
Add technology
 ↓
Hope

Example: Database Index

Suppose:

SELECT *
FROM orders
WHERE user_id = 123
ORDER BY created_at DESC;

The table grows.

Instead of immediately introducing a caching layer, check the query plan.

You might discover that an index solves the problem:

CREATE INDEX idx_orders_user_created
ON orders(user_id, created_at DESC);

Now:

Before:

Query
 ↓
Scan huge table
 ↓
Sort
 ↓
Response

After:

Query
 ↓
Index
 ↓
Relevant rows
 ↓
Response

A one-line index may be better than another distributed component.

Complexity Has Carrying Costs

Every new component introduces:

Operational cost
Debugging cost
Deployment cost
Security cost
Learning cost
Failure modes

Think of architecture as having a complexity budget.

For example:

Database
        1 unit

Redis
        +1

Queue
        +1

Search cluster
        +1

Microservice
        +2

Event bus
        +2

Multi-region
        +3

The numbers aren't literal.

The idea is.

Complexity compounds.

The Diagram

Simple:

Application
    │
    ▼
Database


More complex:

Application
 ├── Redis
 ├── Queue
 ├── Search
 ├── Service A
 ├── Service B
 └── Event Bus
        │
        ▼
     Database

The second architecture may eventually be necessary.

But it should earn its complexity.

Complexity Should Buy You Something

This became a rule I use often:

Every architectural component should solve a measurable problem.

Redis should solve:

Latency
Database load

A queue should solve:

Asynchronous processing
Traffic smoothing
Retryability

A search engine should solve:

Search requirements

Microservices should solve problems like:

Independent deployment
Team ownership
Isolation
Scaling boundaries

Not:

"Big companies use microservices."

5. The Code Review That Taught Me That Readability Is a Performance Feature

The fifth review might have been the most subtle.

The code worked.

It was reasonably fast.

It passed tests.

But the reviewer said something like:

"I understand what this code does, but I don't understand why it does it."

That sentence changed my view of readability.

I had previously thought readability meant:

Good variable names
Short functions
Formatting
Comments

Those things matter.

But deeper readability is about preserving intent.

Code Should Explain the System's Decisions

Consider:

if user.status == "ACTIVE":
    process(user)

Easy to read.

But why?

Maybe inactive users shouldn't be processed.

Or maybe only verified users should.

Or perhaps:

ACTIVE

means something very specific in the domain.

A better design can make the intent explicit:

if user.can_receive_notifications():
    notify(user)

Now the code communicates behavior.

Business Logic Should Read Like Business Logic

Instead of:

if (
    order.status == "PAID"
    and order.created_at < cutoff
    and not order.refunded
):
    ...

you might encapsulate the concept:

if order.is_eligible_for_archive():
    ...

And:

class Order:

    def is_eligible_for_archive(self):
        return (
            self.status == "PAID"
            and self.created_at < cutoff
            and not self.refunded
        )

The complexity didn't disappear.

It moved to a place where it can be named.

That's valuable.

Names Are Compression

A good function name compresses a lot of knowledge.

Consider:

calculate_total()

versus:

calculate_refundable_amount()

The second name tells me more.

Or:

transition_to_paid()

instead of:

update_status("PAID")

The second is a generic mutation.

The first communicates a domain transition.

APIs Should Express Intent

Compare:

PATCH /orders/123

with:

POST /orders/123/cancel

The first exposes implementation.

The second expresses behavior.

This doesn't mean action-oriented endpoints are always better.

The larger lesson is:

Interfaces should communicate the domain's concepts, not merely expose its database.

Implementation: State Transitions

Instead of:

order.status = "CANCELLED"

use:

order.cancel()

Then:

class Order:

    def cancel(self):

        if self.status not in {"PENDING", "PAID"}:
            raise InvalidTransition()

        self.status = "CANCELLED"

Now the domain rule lives with the behavior.

You can test it directly:

def test_paid_order_can_be_cancelled():

    order = Order(status="PAID")

    order.cancel()

    assert order.status == "CANCELLED"

Readability Reduces Future Bugs

This is why I increasingly see readability as a reliability feature.

Suppose an engineer joins the project six months later.

They see:

order.status = "CANCELLED"

They might assume:

Any order can be cancelled.

But if they see:

order.cancel()

they know:

There is probably domain logic here.

The code guides the engineer toward the correct behavior.

That's powerful.

Comments Should Explain Why

Bad comment:

# Increment counter
counter += 1

The code already explains that.

Better:

# Keep this counter monotonic because downstream
# reconciliation uses it to detect missing events.
counter += 1

Now the comment preserves context.

The best comments explain:

Why this exists
Why this approach was chosen
What constraint must not be violated

Not:

What the next line does

The Deeper Pattern Behind All Five Reviews

Looking back, these five reviews seem different.

One was about data modeling.

One was about abstraction.

One was about failure.

One was about complexity.

One was about readability.

But they all point to the same thing.

Engineering is mostly about managing change and uncertainty.

Let's put them together.

Lesson 1: Model Reality Correctly

Reality
  ↓
Domain model
  ↓
Data model
  ↓
Code

If the model is wrong:

Wrong model
  ↓
Correct implementation
  ↓
Correctly implemented mistake

Lesson 2: Don't Abstract Before Understanding

Similarity
    ↓
Observation
    ↓
Understanding
    ↓
Abstraction

Not:

Similarity
    ↓
Generic framework
    ↓
Pain

Lesson 3: Design for Failure

Operation
    ↓
Success
    OR
Failure
    OR
Timeout
    OR
Partial completion
    OR
Unknown state

Good systems account for these.

Lesson 4: Spend Complexity Carefully

Problem
  ↓
Measurement
  ↓
Solution

Not:

Technology
  ↓
Find a problem for it

Lesson 5: Preserve Intent

Code
 ↓
Meaning
 ↓
Future engineer

The goal isn't merely for today's developer to understand the code.

The goal is for someone six months from now to understand it without needing to reconstruct your entire thought process.

Code Review Is Architecture Review

This changed my attitude toward pull requests.

A pull request is not just:

Does this compile?

It's also:

Does this model the domain correctly?

Does this introduce unnecessary coupling?

What happens when it fails?

Will this still make sense in six months?

Does this add complexity we actually need?

Can another engineer understand the intent?

These questions are much more valuable than arguing about tabs versus spaces.

A Practical Code Review Framework

Today, when reviewing code, I like thinking in layers.

Layer 1: Correctness

Does it work?

Check:

Happy path
Edge cases
Invalid input
Failure states
Concurrency

Layer 2: Data

Does the model represent reality?

Ask:

What does this field mean?
What happens when it changes?
What history needs preserving?
What constraints exist?

Layer 3: Architecture

Where does this responsibility belong?

Ask:

Does this create coupling?
Does this violate boundaries?
Is this abstraction justified?

Layer 4: Operations

What happens in production?

Ask:

How do we debug it?
How do we monitor it?
What happens when a dependency fails?
Can we recover?

Layer 5: Future Change

What happens when requirements change?

Ask:

Can this evolve?
Will this force a rewrite?
Does this lock us into a decision too early?

This is where code review starts becoming engineering review.

What I Don't Care About as Much Anymore

Experience changed what I consider important.

I care less about:

One-line functions

and more about:

Correct boundaries

I care less about:

Perfect abstractions

and more about:

Useful abstractions

I care less about:

Maximum performance

and more about:

Measured performance

I care less about:

How clever the implementation looks

and more about:

Whether another engineer can reason about it

And I care much more about:

Failure
Data integrity
Operational visibility

The Best Code Is Often Boring

This is probably one of the biggest changes in my engineering philosophy.

When I was younger, interesting code meant:

Complex algorithm
Clever abstraction
Advanced architecture
New technology

Now, interesting engineering often looks boring.

For example:

def create_order(user, items):

    validate_items(items)

    with transaction():

        order = create_order_record(user)

        create_order_items(order, items)

        return order

There is nothing flashy here.

But if:

validation
transactionality
constraints
authorization
idempotency
observability

are all correctly handled around it, this boring code can be extremely valuable.

Engineering Is Not About Impressing the Compiler

The compiler doesn't care about your architecture.

The database doesn't care about your design philosophy.

The user doesn't care how elegant your dependency injection system is.

Production cares about:

Correctness
Reliability
Performance
Security
Recoverability

And future developers care about:

Clarity
Boundaries
Intent
Changeability

Good engineering balances both.

The Difference Between Code and Software

This distinction became much clearer to me.

Code is what you write.

Software is what happens after you write it.

Software includes:

Code
+
Data
+
Users
+
Infrastructure
+
Failures
+
Deployments
+
Operations
+
Future changes

A code review that only looks at the code is incomplete.

You have to think about the system around it.

A Pull Request Is a Prediction

This is another mental model I like.

Every pull request is a prediction about the future.

When you merge:

This code will behave correctly.

But you're also saying:

This data model will remain useful.

This boundary will make sense.

This complexity is worth it.

These failure modes are acceptable.

Future developers will understand this.

That's a much bigger claim.

Code review is the process of challenging that prediction.

The Five Questions I Ask Now

If I had to reduce everything I've learned into five questions, they would be:

1. What reality is this code modeling?

If I don't understand the underlying concept, I shouldn't approve the implementation yet.

2. What happens when this fails halfway?

Timeouts, retries, crashes and duplicate requests are normal parts of production.

3. Is this abstraction actually earned?

Similarity isn't enough.

4. What complexity am I introducing?

Every abstraction, dependency and service has a maintenance cost.

5. Will the next engineer understand why?

Because eventually, someone else will have to change it.

And that person might be me six months later.

A Complete Review Example

Imagine this pull request:

def process_payment(order):

    result = provider.charge(order.total)

    if result.success:
        order.status = "PAID"
        db.save(order)

    return result

A shallow review says:

Looks good.

A deeper review asks:

1. What if provider succeeds and db.save() fails?

2. Can this function be called twice?

3. Is provider.charge() idempotent?

4. Where is the payment ID stored?

5. What if the provider times out?

6. How do we reconcile unknown states?

7. Is order.status allowed to change directly?

8. Is payment amount derived from trusted data?

9. How do we observe failures?

10. How do we test duplicate requests?

Now we're reviewing a system.

A More Robust Version

Conceptually:

def process_payment(order, idempotency_key):

    payment = payment_repo.find_by_key(
        idempotency_key
    )

    if payment:
        return payment

    payment = payment_repo.create(
        order_id=order.id,
        amount_cents=order.total,
        idempotency_key=idempotency_key,
        status="PENDING"
    )

    try:

        result = provider.charge(
            amount=payment.amount_cents,
            idempotency_key=idempotency_key
        )

        if result.success:
            payment.mark_paid(
                provider_id=result.id
            )
        else:
            payment.mark_failed()

        payment_repo.save(payment)

        return payment

    except TimeoutError:

        payment.mark_unknown()

        payment_repo.save(payment)

        raise

This still isn't a complete payment system.

Real systems need reconciliation, webhooks, retries, provider-specific behavior and more.

But notice how different the mental model is.

We're explicitly modeling:

Identity
State
Failure
Retries
External systems
Historical records

That's engineering.

Code Review Should Teach, Not Just Judge

Another lesson I learned is that the best code reviews don't simply say:

"Wrong."

They explain:

"Here's the risk."

For example, instead of:

Don't do this.

say:

This couples order state directly to the payment provider response.
If the provider succeeds but this transaction fails, our local state
can remain PENDING. Consider modeling the payment state separately and
using an idempotency key so retries don't create duplicate charges.

That teaches a principle.

The developer can use it elsewhere.

That's a much better review.

From Reviewer to Better Engineer

Eventually, code review stops being something you do to other people's code.

You start reviewing your own code mentally before opening the pull request.

You start thinking:

Will someone question this abstraction?
What happens if this times out?
Is this data historical?
Do I need this dependency?
Can I explain why this exists?

That's when code review has done its job.

It has changed how you think.

Final Thoughts

The five code reviews I remember most weren't necessarily the ones that caught the biggest bugs.

They were the ones that changed the questions I asked.

The first taught me:

Correctness starts with modeling reality.

The second taught me:

Abstraction is a tool, not a virtue.

The third taught me:

Failure is not an edge case. It is part of the normal execution environment.

The fourth taught me:

Complexity is a budget. Spend it where the problem demands it.

The fifth taught me:

Readable code preserves intent for people you haven't met yet.

Together, these lessons changed my definition of engineering.

I used to think engineering was mostly about making software work.

Now I think it's about making software continue to work as reality changes.

Users change.

Requirements change.

Traffic changes.

Databases grow.

Dependencies fail.

Teams change.

Business models change.

The original developer leaves.

And eventually, somebody opens the code six months later and asks:

"Why did they build it this way?"

The quality of your engineering is partly determined by how easily that person can answer that question.

That's why I don't think the best code is necessarily the shortest code.

Or the cleverest code.

Or the most abstract code.

Or even the fastest code.

The best code is code that makes the system's important truths visible.

It tells you:

What the system believes.
What the system guarantees.
What can fail.
Where responsibility lives.
Why a decision was made.

And when those things are clear, something interesting happens.

The code becomes easier to change.

The architecture becomes easier to reason about.

The bugs become easier to diagnose.

The reviews become more meaningful.

And the system starts feeling less like a pile of instructions and more like an engineered machine.

That's probably the biggest thing code reviews have taught me.

Good engineering isn't about writing code that looks impressive.

It's about building systems that remain understandable when the original assumptions start breaking.

And eventually, they always do.

The "Minimum Viable System": How to Ship Early Without Shipping Junk

Derek mwale — Fri, 07 Aug 2026 05:57:56 +0000

There is a dangerous piece of advice in software development:

"Just build an MVP."

It sounds reasonable.

Build the smallest possible version.

Launch quickly.

Get feedback.

Improve later.

The problem is that many developers interpret "minimum viable product" as:

Remove everything.
Skip architecture.
Ignore security.
Ignore testing.
Hardcode everything.
Ship it.

Then six months later, the product is technically alive but practically dead.

Every new feature breaks something.

Every database migration is terrifying.

Every customer request requires a workaround.

The codebase has become a collection of decisions that made sense individually but make no sense together.

The team spends more time fighting the system than improving the product.

This is where I think the idea of a Minimum Viable System becomes much more useful.

The goal isn't to build the smallest amount of software possible.

The goal is to build the smallest system that can safely support learning.

That distinction matters.

A Minimum Viable System is not junk software.

It is intentionally incomplete software with enough engineering discipline to survive contact with reality.

The system doesn't need every feature.

It doesn't need perfect scalability.

It doesn't need a distributed architecture.

It doesn't need Kubernetes.

It doesn't need twelve microservices.

But it does need a few things to be fundamentally correct.

It needs a coherent data model.

It needs clear boundaries.

It needs basic observability.

It needs reasonable security.

It needs a deployment process.

It needs tests around important behavior.

And most importantly, it needs to be designed so that the next version doesn't require throwing away the first one.

That is the difference between shipping early and shipping junk.

The Problem With "Move Fast"

Software engineers often imagine speed as:

Idea
 ↓
Code
 ↓
Deploy

But real software doesn't work that way.

The moment users arrive, the system becomes a living thing.

                USERS
                  │
                  ▼
              PRODUCT
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
    Traffic     Data       Feedback
       │          │          │
       └──────────┼──────────┘
                  ▼
              ENGINEERING
                  │
                  ▼
               CHANGE
                  │
                  ▼
               PRODUCT

Every deployment creates new constraints.

Every customer creates new data.

Every integration creates another dependency.

Every feature creates another behavior.

The software becomes increasingly difficult to change.

So the real objective of early development should not be:

"How quickly can I write the first version?"

It should be:

"How quickly can I create a system that teaches me what I need to know without creating unnecessary future problems?"

That is a much better question.

MVP vs Minimum Viable System

Let's make the distinction explicit.

A traditional MVP often gets described as:

Minimum
+
Viable
+
Product

I would describe a Minimum Viable System as:

Minimum
+
Useful
+
Safe
+
Observable
+
Changeable
+
Deployable
+
System

The product is what users experience.

The system is everything required to make that experience reliable.

For example, suppose you're building a simple marketplace.

The MVP might be:

User
 ↓
Browse Products
 ↓
Add to Cart
 ↓
Checkout

But the Minimum Viable System is:

                   Marketplace
                       │
       ┌───────────────┼───────────────┐
       ▼               ▼               ▼
   Frontend         Backend         Database
                       │
             ┌─────────┼─────────┐
             ▼         ▼         ▼
          Auth       Orders    Payments
                       │
                       ▼
                  Observability
                       │
                       ▼
                   Deployment

You don't need everything.

But the things you do build need to form a coherent system.

The First Principle: Remove Features, Not Engineering

This is probably the most important idea in the entire article.

When you're trying to ship early, remove scope.

Don't automatically remove engineering discipline.

For example, you might decide:

No social login.
No mobile application.
No recommendation engine.
No advanced analytics.
No multi-language support.
No marketplace messaging.

Good.

That's scope reduction.

But don't decide:

No authentication.
No validation.
No database constraints.
No backups.
No error handling.
No logs.

That's not scope reduction.

That's risk accumulation.

A useful way to think about it is:

                 PRODUCT SCOPE
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          Remove               Keep
          Features             Foundations
             │                   │
             ▼                   ▼
        Faster Launch        Safer System

The trick is knowing which is which.

What Can Be Cheap?

Almost everything visible to the user can be simplified.

Suppose you're building a dashboard.

You don't need:

100 chart types
Real-time WebSockets
Custom themes
AI analytics
Advanced filtering
Drag-and-drop layouts

You might only need:

Three metrics
One table
One chart
One filter

That's perfectly reasonable.

But your backend should still have:

Input validation
Authentication
Authorization
Database constraints
Error handling
Logging

The interface can be primitive.

The foundations should be intentional.

Think in System Invariants

One of the best ways to build a small system without building a fragile system is to define invariants.

An invariant is something that should always be true.

For an e-commerce system:

An order must belong to a user.

An order must contain at least one item.

An order item must reference a valid product.

A paid order cannot become unpaid accidentally.

A product price must not be negative.

These rules are more important than whether the interface has a beautiful animation.

Example: Database Invariants

Suppose we have:

CREATE TABLE products (
    id BIGSERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    price_cents INTEGER NOT NULL
);

We can strengthen the system:

CREATE TABLE products (
    id BIGSERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    price_cents INTEGER NOT NULL
        CHECK (price_cents >= 0)
);

Now the database itself protects an invariant.

This is important because your application code isn't the only thing touching your data forever.

Eventually you'll have:

Backend
Admin panel
Scripts
Workers
Imports
Migrations
Analytics jobs

Database constraints are a final line of defense.

Don't Build a Cathedral

One of the biggest mistakes I've seen in early software development is architecture designed for imaginary scale.

The developer thinks:

"What if we get ten million users?"

So they build:

API Gateway
 ↓
Auth Service
 ↓
User Service
 ↓
Product Service
 ↓
Order Service
 ↓
Event Bus
 ↓
Kafka
 ↓
Multiple databases
 ↓
Redis cluster
 ↓
Kubernetes
 ↓
Service mesh

The product has:

12 users.

This is architecture fiction.

The architecture should reflect the problem you actually have.

Start With the Boring Architecture

For many products, I would start with:

                 Internet
                    │
                    ▼
                 Nginx
                    │
                    ▼
              Application
                    │
              ┌─────┴─────┐
              ▼           ▼
           Database      Redis

Maybe you don't even need Redis.

Maybe:

Internet
   ↓
Application
   ↓
PostgreSQL

is enough.

There is nothing embarrassing about this.

A well-designed monolith can take you surprisingly far.

The goal is not architectural complexity.

The goal is architectural clarity.

A Good Monolith Has Boundaries

A monolith doesn't mean:

Everything everywhere.

It can still have internal modules.

For example:

app/
├── users/
├── products/
├── orders/
├── payments/
├── notifications/
└── shared/

The application is one deployable unit.

But the concepts are separated.

             Application
                  │
     ┌────────────┼────────────┐
     ▼            ▼            ▼
   Users       Products      Orders
                                │
                                ▼
                            Payments

This is a very powerful starting point.

If one day the order subsystem genuinely needs to become independent, you already have a conceptual boundary.

Implementation: A Small Modular Backend

Here's a simplified Python-style structure:

class UserService:
    def create_user(self, email, password):
        ...

class ProductService:
    def create_product(self, name, price):
        ...

class OrderService:
    def create_order(self, user_id, items):
        ...

class PaymentService:
    def charge(self, order_id, amount):
        ...

The services can live inside one application.

You don't need a network boundary between them yet.

That is an important distinction:

Logical boundary
≠
Network boundary

Create the logical boundary first.

Introduce distributed infrastructure only when the problem demands it.

The Minimum Viable Data Model

Early developers sometimes focus heavily on screens.

I usually think about the data first.

Ask:

What entities exist?

What relationships exist?

What must always be true?

What changes over time?

What needs history?

For a marketplace:

User
 │
 └────< Order
           │
           └────< OrderItem >──── Product

A simple relational model might be:

CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    email TEXT NOT NULL UNIQUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE products (
    id BIGSERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    price_cents INTEGER NOT NULL
        CHECK (price_cents >= 0),
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE orders (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL
        REFERENCES users(id),
    status TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE order_items (
    id BIGSERIAL PRIMARY KEY,
    order_id BIGINT NOT NULL
        REFERENCES orders(id),
    product_id BIGINT NOT NULL
        REFERENCES products(id),
    quantity INTEGER NOT NULL
        CHECK (quantity > 0),
    price_cents INTEGER NOT NULL
        CHECK (price_cents >= 0)
);

Notice something important.

This is not over-engineered.

It's just explicit.

Why Store the Price on Order Items?

Someone might ask:

"Why not always read the current product price?"

Because historical data matters.

Suppose:

Product today: $20
Product yesterday: $15

A customer bought it yesterday.

The order should still represent:

$15

not:

Current product price = $20

This is a small modeling decision.

But these are the kinds of decisions that separate durable software from fragile software.

Build the Smallest Correct Workflow

Let's say we need checkout.

Don't build:

Coupons
Gift cards
Subscriptions
Loyalty points
Multiple currencies
Installments
Wallets

Start with:

Cart
 ↓
Create Order
 ↓
Calculate Total
 ↓
Payment
 ↓
Mark Paid

The workflow should be correct.

                 Checkout
                    │
                    ▼
              Validate Cart
                    │
                    ▼
              Create Order
                    │
                    ▼
             Calculate Total
                    │
                    ▼
              Process Payment
                    │
             ┌──────┴──────┐
             ▼             ▼
          Success        Failure
             │             │
             ▼             ▼
        Mark Paid       Mark Failed

This is enough to learn.

Transactions Matter Early

Imagine this:

Create order
   ↓
Payment succeeds
   ↓
Server crashes
   ↓
Order isn't marked paid

Now the external payment provider says:

PAID

Your database says:

PENDING

You have created an inconsistency.

You don't need microservices to encounter distributed systems problems.

They arrive naturally.

Use Transactions Where They Make Sense

For local database operations:

with db.transaction():

    order = create_order(user_id)

    for item in items:
        create_order_item(order.id, item)

    update_order_total(order.id, total)

If something fails:

Operation
   ↓
Failure
   ↓
Rollback

instead of:

Order created
Order item 1 created
Order item 2 created
Order item 3 failed

Partial state is one of the easiest ways to create early technical debt.

But Don't Pretend External APIs Are Transactional

A payment provider isn't your database.

This won't magically work:

with db.transaction():

    order = create_order()

    payment_provider.charge()

    mark_paid()

If the payment succeeds and your transaction rolls back, the payment still happened.

This is where idempotency and state machines become important.

Build Explicit State Machines

Instead of random booleans:

is_paid
is_cancelled
is_refunded

use a clear state:

PENDING
PAID
FAILED
CANCELLED
REFUNDED

And define valid transitions:

              ┌─────────┐
              │ PENDING │
              └────┬────┘
             ┌─────┴─────┐
             ▼           ▼
           PAID        FAILED
             │
             ▼
          REFUNDED

Then implementation becomes explicit:

VALID_TRANSITIONS = {
    "PENDING": {"PAID", "FAILED", "CANCELLED"},
    "PAID": {"REFUNDED"},
    "FAILED": set(),
    "CANCELLED": set(),
    "REFUNDED": set(),
}

Now your system has a model of reality.

Idempotency: One of the Cheapest High-Value Investments

Imagine the user clicks:

Pay

The request reaches the server.

The payment succeeds.

The response gets lost.

The client retries.

Now you might charge them twice.

This is why idempotency matters.

A request can include:

Idempotency-Key: 7b1a-92f4

The backend stores the result:

7b1a-92f4
     ↓
Payment #9281
     ↓
SUCCESS

If the request arrives again:

Same key
   ↓
Existing result
   ↓
Return previous result

No duplicate payment.

This is not premature optimization.

It's correctness.

The Minimum Viable API

Don't build 100 endpoints.

Start with the smallest workflow.

For a marketplace:

POST /auth/register
POST /auth/login

GET /products
GET /products/:id

POST /orders
GET /orders/:id

POST /payments

That's enough to launch a surprisingly functional system.

But each endpoint should have:

Validation
Authentication
Authorization
Consistent errors
Logging
Tests

API Error Design

Avoid this:

{
  "error": "Something went wrong"
}

Prefer structured errors:

{
  "error": {
    "code": "INVALID_ORDER",
    "message": "Order contains no items",
    "request_id": "req_82a91"
  }
}

Now the client can understand what happened.

The request ID also helps debugging.

Implementation: Request IDs

A simple middleware pattern:

import uuid

def request_id_middleware(request, next_handler):

    request_id = request.headers.get(
        "X-Request-ID"
    ) or str(uuid.uuid4())

    response = next_handler(request)

    response.headers["X-Request-ID"] = request_id

    return response

Then logs can contain:

request_id=req_82a91

Now one customer issue can be traced across the system.

Observability Is Not a Luxury

You don't need a massive observability platform on day one.

But you need to know:

What happened?
When?
To whom?
Where?
Why?

At minimum:

Logs
Errors
Request IDs
Basic metrics

A simple log:

logger.info(
    "order_created",
    extra={
        "order_id": order.id,
        "user_id": user.id,
        "request_id": request_id
    }
)

Now you have a trail.

The Minimum Viable Metrics

I would start with:

Requests
Errors
Latency
Active users
Successful transactions
Failed transactions

That's enough to answer many early questions.

For example:

Requests:       12,430
Errors:             83
Error rate:       0.67%
P95 latency:       420ms
Orders:             218
Payments failed:     11

This is far more useful than a dashboard containing 200 meaningless metrics.

Testing: Don't Test Everything

Another misunderstanding of MVP development is:

"We don't have time for tests."

You don't have time for the wrong tests.

You don't need hundreds of tests for every CSS component.

Test the business invariants.

For example:

def test_order_requires_items():
    with pytest.raises(ValueError):
        create_order(user_id=1, items=[])

Test:

def test_negative_price_is_rejected():
    with pytest.raises(ValueError):
        create_product(
            name="Laptop",
            price_cents=-100
        )

Test state transitions:

def test_paid_order_can_be_refunded():

    order = Order(status="PAID")

    order.transition("REFUNDED")

    assert order.status == "REFUNDED"

And invalid transitions:

def test_pending_order_cannot_be_refunded():

    order = Order(status="PENDING")

    with pytest.raises(ValueError):
        order.transition("REFUNDED")

These tests protect the system's meaning.

Test the Expensive-to-Fix Things

A useful rule:

Test behavior that would be expensive or embarrassing to break.

Examples:

Authentication
Payments
Permissions
Data integrity
Orders
Billing
State transitions
Important calculations

Don't obsess over achieving:

100% code coverage

while missing:

100% correctness of money movement

Coverage is a metric.

Correctness is the goal.

Security Must Exist Before Users

One of the worst MVP philosophies is:

"We'll add security after we validate the idea."

Users don't care whether you're validating an idea.

If their passwords are leaked, they're still leaked.

The minimum system should include:

HTTPS
Password hashing
Authentication
Authorization
Input validation
Secure secrets
Database access controls
Rate limiting where necessary
Dependency updates

You don't need a giant security department.

You do need basic security hygiene.

Never Store Plaintext Passwords

This should be non-negotiable.

Don't:

user.password = password

Use a password hashing algorithm through a reputable library.

Conceptually:

password_hash = hash_password(password)

verify_password(
    password,
    password_hash
)

The database should never contain the original password.

Authorization Is Different From Authentication

Authentication asks:

Who are you?

Authorization asks:

What are you allowed to do?

For example:

User A
 ↓
GET /orders/123

The system shouldn't simply check:

Is User A logged in?

It should check:

Does Order 123 belong to User A?

A simple rule:

if order.user_id != current_user.id:
    raise Forbidden()

Many security vulnerabilities are simply authorization mistakes.

Don't Build Your Own Cryptography

This is another place where "minimum viable" can become dangerous.

Don't implement:

Your own password hashing
Your own encryption algorithm
Your own JWT implementation
Your own TLS

Use established libraries.

Build your product.

Not your own cryptographic primitive.

Deployment Is Part of the Product

A system that works on your laptop isn't a product.

This:

Works on my machine

is not deployment.

A minimum viable system needs a repeatable path:

Git
 ↓
Build
 ↓
Test
 ↓
Deploy
 ↓
Health Check

Even if you're the only developer.

A Minimal Deployment Pipeline

                 Git Push
                    │
                    ▼
                  CI
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
       Tests              Build
          │                   │
          └─────────┬─────────┘
                    ▼
                 Deploy
                    │
                    ▼
               Health Check
                    │
              ┌─────┴─────┐
              ▼           ▼
           Healthy      Failed
              │           │
              ▼           ▼
            Live       Rollback

This is enough.

You don't need an elaborate platform team.

Health Checks

A simple endpoint:

GET /health

Response:

{
  "status": "ok"
}

But a deeper readiness check might verify:

Database
Redis
Critical dependencies

For example:

{
  "status": "ready",
  "database": "ok"
}

This becomes extremely useful during deployments.

Feature Flags Are Extremely Powerful

You don't always have to choose:

Build feature
OR
Don't build feature

You can build:

Feature
 ↓
Disabled
 ↓
Deploy
 ↓
Enable for developers
 ↓
Enable for 5%
 ↓
Enable for 50%
 ↓
Enable for everyone

Example:

if feature_flags.enabled(
    "new_checkout",
    user_id
):
    return new_checkout()

return old_checkout()

Now deployment and release become separate concepts.

This gives you much more control.

The Minimum Viable System Should Be Observable Before It Is Scaled

This is a principle I strongly believe in.

Don't scale something you can't understand.

Suppose your application gets slow.

If you have:

No logs
No metrics
No tracing
No profiling

you don't know why.

You start guessing.

Then you add:

Redis

It doesn't fix the problem.

Then:

More servers

Still doesn't fix it.

Then:

Database replica

Still broken.

You are scaling blind.

Measure Before Optimizing

A simple workflow:

Problem
  ↓
Measure
  ↓
Identify bottleneck
  ↓
Change
  ↓
Measure again

Not:

Problem
 ↓
Add technology
 ↓
Hope

The Database Is Often Your First Scaling Problem

Suppose your query is:

SELECT *
FROM orders
WHERE user_id = 42
ORDER BY created_at DESC;

If the table grows, indexing matters.

CREATE INDEX idx_orders_user_created
ON orders(user_id, created_at DESC);

This is often more valuable than introducing another service.

A good database design can eliminate entire categories of infrastructure complexity.

Don't Prematurely Cache Everything

Caching is useful.

But it introduces invalidation problems.

Start with:

Database

Measure.

If a query becomes expensive:

Optimize query
 ↓
Add index
 ↓
Measure
 ↓
Cache if necessary

Not:

Everything → Redis

The Minimum Viable System Has a Debt Budget

Technical debt isn't automatically bad.

Sometimes debt is intentional.

For example:

We use a simple email implementation now.
We'll build a notification service later.

That's reasonable.

But record the debt.

Create a simple document:

TECHNICAL DEBT

1. Replace temporary email provider
2. Add stronger rate limiting
3. Refactor reporting query
4. Improve background job retries

Now debt is visible.

Invisible debt is much more dangerous than documented debt.

Distinguish Good Debt From Bad Debt

Good debt

Simple architecture
Temporary UI
Manual admin process
Limited feature set
Basic deployment

These can be improved later.

Bad debt

Security vulnerability
Corruptible data
Unclear ownership
No backups
No migration strategy
Duplicated business logic everywhere

These become exponentially more expensive.

A Useful Equation

I think about early software roughly like this:

Future Change Cost
≈
Complexity
×
Coupling
×
Uncertainty

You can't eliminate uncertainty.

You're building something new.

But you can reduce:

Complexity
Coupling

That's what good MVP architecture does.

Build for Change, Not for Scale

This distinction is extremely important.

Early products usually don't need:

10 million requests/second

They need:

The ability to change direction quickly.

You might discover that users don't want feature A.

They want feature B.

If your architecture is easy to change:

A → B

is manageable.

If your architecture is tightly coupled:

A
 ↓
Service 1
 ↓
Service 2
 ↓
Service 3
 ↓
Database
 ↓
Event bus
 ↓
Service 4

changing A can become a project of its own.

The Real Enemy Is Coupling

Consider this:

def create_order():
    send_email()
    charge_payment()
    update_inventory()
    create_analytics_event()
    notify_admin()

One function now knows everything.

That's coupling.

A cleaner approach:

def create_order():
    order = order_service.create()
    events.publish(
        OrderCreated(order.id)
    )
    return order

Then other systems can react.

Even inside a monolith, this creates cleaner boundaries.

But Don't Turn Everything Into Events

Again, balance.

This:

create order
 ↓
event
 ↓
handler
 ↓
event
 ↓
handler
 ↓
event

can become difficult to reason about.

Use events where asynchronous behavior genuinely helps.

For example:

OrderCreated
 ↓
Send email
 ↓
Update analytics

Those operations don't necessarily need to block checkout.

But:

Calculate order total

should probably remain part of the immediate transaction.

Synchronous vs Asynchronous

A useful early rule:

Synchronous

Use when the user needs the result immediately.

Create order
Calculate total
Validate permissions

Asynchronous

Use when the work can happen later.

Send email
Generate report
Resize image
Process analytics

This keeps the core workflow simple.

Background Jobs Without Overengineering

You might start with:

Application
 ↓
Database

Then introduce a simple queue when needed:

Application
 ↓
Queue
 ↓
Worker

For example:

jobs.enqueue(
    send_order_confirmation,
    order.id
)

The worker handles:

def send_order_confirmation(order_id):

    order = get_order(order_id)

    email.send(
        to=order.user.email,
        template="order_confirmation"
    )

Now the checkout request doesn't wait for email delivery.

Retry Carefully

Background jobs can fail.

Use:

Attempt 1
 ↓
Failure
 ↓
Attempt 2
 ↓
Failure
 ↓
Attempt 3
 ↓
Dead Letter Queue

But make the job idempotent.

Otherwise:

Retry
 ↓
Duplicate action

can become a serious problem.

The Minimum Viable Admin System

This is something developers often overlook.

You need a way to operate your own product.

At minimum:

View users
View orders
View errors
View important records
Disable problematic accounts
Retry failed jobs

You don't need a beautiful admin dashboard.

A basic internal interface can save enormous amounts of time.

Build Manual Processes Before Automating Them

Suppose you're unsure whether refunds should be automatic.

Don't spend two weeks building refund automation.

Start with:

Customer requests refund
 ↓
Admin reviews
 ↓
Admin clicks refund

Once you have enough volume:

Automated Refund

The manual process teaches you the rules.

This is one of the most powerful forms of product discovery.

Automation Should Follow Understanding

The sequence should often be:

Manual
 ↓
Understand
 ↓
Standardize
 ↓
Automate

Not:

Unknown Process
 ↓
Automate
 ↓
Discover Problems

Automation amplifies understanding.

It also amplifies confusion.

The Minimum Viable System Is a Learning Machine

This is the deeper idea.

You're not merely building:

Product

You're building:

Product
+
Feedback Loop

The system should teach you.

                    USERS
                      │
                      ▼
                   PRODUCT
                      │
             ┌────────┼────────┐
             ▼        ▼        ▼
          Metrics   Feedback   Errors
             │        │        │
             └────────┼────────┘
                      ▼
                  INSIGHTS
                      │
                      ▼
                    CHANGE
                      │
                      ▼
                   PRODUCT

This is why observability, analytics and customer feedback matter so much.

They aren't side projects.

They are part of the learning system.

What I Would Build in the First Two Weeks

If I were building a new backend-heavy product, my initial system might look like this.

Day 1–2: Domain

Define:

Users
Core entities
Relationships
Important states
Business rules

Draw the data model.

User
 │
 ├── Orders
 │     │
 │     └── Items
 │
 └── Profile

Day 3–4: Database

Implement:

Tables
Primary keys
Foreign keys
Constraints
Indexes
Migrations

Don't obsess over optimization.

Make the model correct.

Day 5–6: Core API

Implement only:

Authentication
Core resource creation
Core resource retrieval
Primary workflow

No 50-endpoint REST API.

Day 7: Tests

Test:

Core business rules
Important failure cases
Authorization
Data invariants

Day 8: Deployment

Create:

CI
Production environment
Environment variables
Database migrations
Health check

Day 9: Observability

Add:

Structured logs
Request IDs
Error tracking
Basic metrics

Day 10: Launch

Give it to real users.

Not after six months.

Now.

Then Let Reality Drive Development

After launch, don't immediately build 30 more features.

Watch.

Ask:

Where do users struggle?

What do they actually use?

Where do they abandon the workflow?

What breaks?

What do they request repeatedly?

What are they willing to pay for?

Then build.

The Product Roadmap Should Come From Evidence

A bad roadmap:

AI
Blockchain
Mobile app
Social feed
NFTs
Recommendations
Gamification

A better roadmap:

Users cannot complete checkout
 ↓
Fix checkout

Users repeatedly search for invoices
 ↓
Add invoices

Customers ask for export
 ↓
Add CSV export

The system tells you where to go.

Don't Confuse "Small" With "Cheap"

A small system can still be high quality.

For example:

One backend
One database
One frontend
One deployment

can be extremely well engineered.

And:

12 services
Kubernetes
Kafka
Redis
GraphQL
Service mesh

can be terrible.

Architecture quality is not proportional to architecture size.

The Smallest Good System

If I had to reduce the idea of a Minimum Viable System to one architecture, it would be:

                 USERS
                   │
                   ▼
                FRONTEND
                   │
                   ▼
                API
                   │
          ┌────────┼────────┐
          ▼        ▼        ▼
       Domain    Auth     Validation
       Logic
          │
          ▼
       Database
          │
          ▼
        Migrations

Supporting:
    Logs
    Metrics
    Tests
    Backups
    CI/CD

That's it.

No Kubernetes.

No microservices.

No event streaming platform.

No AI agent swarm.

No infrastructure theater.

Just a coherent system.

The Minimum Viable System Checklist

Before launching, I would ask:

Product

[ ] Is the core user problem clear?
[ ] Can a user complete the main workflow?
[ ] Is there a measurable outcome?

Data

[ ] Are relationships modeled correctly?
[ ] Are important constraints enforced?
[ ] Are migrations versioned?
[ ] Are backups configured?

Backend

[ ] Are APIs validated?
[ ] Are errors structured?
[ ] Are permissions enforced?
[ ] Are important operations transactional?

Security

[ ] Are passwords hashed?
[ ] Is HTTPS enabled?
[ ] Are secrets outside source control?
[ ] Is authorization checked?
[ ] Are dangerous inputs validated?

Reliability

[ ] Are errors logged?
[ ] Are request IDs available?
[ ] Is there a health endpoint?
[ ] Can failed jobs be retried?

Deployment

[ ] Can the system be deployed repeatably?
[ ] Can database migrations run safely?
[ ] Is rollback possible?

Product Learning

[ ] Can we measure usage?
[ ] Can users give feedback?
[ ] Can we identify failures?
[ ] Can we understand why users leave?

If these answers are mostly yes, you probably have something much more valuable than a typical MVP.

You have a Minimum Viable System.

What I Would Not Build

At the beginning, I would probably avoid:

Microservices
Kubernetes
Custom authentication protocol
Custom payment infrastructure
Complex event-driven architecture
Premature caching
Multi-region deployment
Multiple databases
Distributed tracing infrastructure
AI agents
Complex recommendation engines

Unless the actual problem requires them.

Not because these technologies are bad.

They're excellent technologies.

But technology should answer a problem.

It shouldn't become the problem.

The "Future Scale" Test

Before adding infrastructure, ask three questions:

1. What problem am I solving?

Not:

"Would Redis be cool?"

But:

"Which measured bottleneck requires Redis?"

2. Do I actually have the problem?

Not:

"What if we have millions of users?"

But:

"Do our current measurements indicate that the database is the bottleneck?"

3. What's the simplest solution?

Maybe the answer is:

Add an index.

Not:

Create another distributed service.

This discipline can save months.

The Most Valuable Feature Is Sometimes "Nothing"

This is something I learned to appreciate more with experience.

A developer sees an empty space in the product and thinks:

"We should build something here."

But sometimes the best decision is:

Don't build it.

Every feature adds:

Code
Tests
Documentation
UI
Data
Support
Security
Maintenance

The cost isn't just development.

It's permanent ownership.

Every Feature Becomes a Contract

Once users depend on something, removing it becomes difficult.

For example:

Feature
 ↓
Users depend on it
 ↓
Data accumulates
 ↓
Documentation exists
 ↓
Integrations appear
 ↓
Support depends on it

Now the feature is part of the system.

This is why early product development should be ruthless about scope.

You're not just choosing what to build.

You're choosing what you will have to maintain.

Software Is a Long-Term Commitment

Writing code is easy compared with maintaining code.

A hundred lines today might become:

100 lines
 ↓
500 lines
 ↓
3 developers
 ↓
10 integrations
 ↓
100 customers
 ↓
Production dependency

This is why I care about the first architecture.

Not because it needs to be perfect.

But because it establishes the direction.

The First Version Is a Seed

I like thinking about software like a seed.

You don't need to build the entire tree.

But you shouldn't plant something that is fundamentally malformed.

A good seed has:

Clear structure
Healthy boundaries
Correct foundations
Room to grow

The first version of your product should work the same way.

               VERSION 1
                   │
          ┌────────┼────────┐
          ▼        ▼        ▼
       Users      Data     Logic
          │        │        │
          └────────┼────────┘
                   ▼
                Feedback
                   │
                   ▼
               VERSION 2
                   │
                   ▼
               VERSION 3
                   │
                   ▼
              REAL SYSTEM

You don't know what version 10 will look like.

That's okay.

Your job is to make version 1 good enough to discover version 2.

Shipping Early Is About Managing Uncertainty

This is the part of MVP thinking that I think gets lost.

You're not shipping early because you don't care about quality.

You're shipping early because you don't yet know enough to justify building everything.

That's a completely different philosophy.

High uncertainty
      +
Small system
      ↓
Fast feedback
      ↓
More knowledge
      ↓
Better architecture
      ↓
Better product

The mistake is thinking:

High uncertainty
      +
Huge system
      ↓
Success

Usually you just get an expensive guess.

The Best Early Architecture Is a Feedback Architecture

This is ultimately what I mean by Minimum Viable System.

Build enough to:

Create value
Collect data
Observe behavior
Protect users
Change quickly

Everything else can wait.

Your architecture should make the feedback loop fast.

                  IDEA
                   │
                   ▼
                BUILD
                   │
                   ▼
                SHIP
                   │
                   ▼
                USERS
                   │
                   ▼
                MEASURE
                   │
                   ▼
                LEARN
                   │
                   ▼
                CHANGE
                   │
                   └───────────────► BUILD

That's the real engine of an early-stage product.

Final Thoughts

I no longer think the goal of early software development should be to build an MVP.

The phrase has become too easy to misunderstand.

The goal should be to build a Minimum Viable System.

A system that is small enough to build quickly.

Simple enough to understand.

Reliable enough to trust.

Secure enough to expose to users.

Observable enough to debug.

Flexible enough to change.

And incomplete enough to leave room for discovery.

The philosophy is simple:

Reduce Scope
     ↓
Keep Foundations
     ↓
Ship
     ↓
Observe
     ↓
Learn
     ↓
Improve

Don't remove engineering just because the product is young.

Remove unnecessary product complexity.

Don't build for imaginary millions of users.

Build for the users you actually have.

Don't build ten services because you might need them someday.

Build one good system with boundaries.

Don't automate processes you don't understand.

Do them manually until the pattern becomes obvious.

Don't test every line of code.

Test the rules that define whether your product is correct.

Don't optimize everything.

Measure first.

Don't aim for perfect architecture.

Aim for architecture that can evolve.

And most importantly:

Don't confuse shipping early with shipping carelessly.

A Minimum Viable System is not an excuse for bad engineering.

It is a strategy for focused engineering.

You are deliberately saying:

"We don't know everything yet, so we won't build everything yet."

But you're also saying:

"The things we do build will have foundations strong enough to survive what we learn."

That is the balance.

Because the real goal of version one isn't to prove that you can write software.

It's to create enough reality that the software can teach you what to build next.

And that's the difference between a prototype that gets thrown away...

and a small system that eventually becomes a real product.

5 Emerging Tech Trends Quietly Reshaping the Enterprise

Derek mwale — Fri, 07 Aug 2026 05:49:00 +0000

Technology doesn't always change the enterprise with a dramatic announcement.

Sometimes the biggest transformation happens quietly.

A few engineers start experimenting with something.

A small team deploys it internally.

A developer builds a prototype over a weekend.

A security team begins writing policies around it.

Then suddenly, three years later, the architecture of the company looks completely different.

I think we are entering one of those periods.

The obvious story is artificial intelligence.

But "AI" is too broad to describe what is actually happening.

The deeper transformation is architectural.

AI is moving from chat interfaces into workflows.

Software is becoming increasingly autonomous.

Compute is moving closer to where data is generated.

Sensitive workloads are being redesigned around new trust models.

Enterprise infrastructure is being rebuilt around AI-native applications rather than traditional web applications.

And physical machines are beginning to inherit capabilities that previously existed only in software.

Several major technology research organizations are pointing toward the same general direction. Gartner's 2026 technology trends include multiagent systems, domain-specific language models, physical AI, confidential computing, AI-native development platforms and AI security platforms. Deloitte similarly describes a shift toward agentic workforces, physical AI, new AI infrastructure and security architectures.

What interests me isn't the hype.

It's the engineering underneath it.

Because the enterprise of the future isn't simply going to have "more AI."

It is going to have a fundamentally different architecture.

In this article, I'll explore five technologies that I believe are quietly pushing enterprise software in that direction:

Agentic AI and autonomous software
AI-native infrastructure and the new compute stack
Confidential computing and trust-by-design
Edge AI and physical intelligence
Domain-specific AI and smaller intelligent systems

The important part is not merely understanding what these technologies are.

It's understanding what they do to the way we build software.

1. Agentic AI: Software That Doesn't Just Respond

For decades, enterprise software followed a relatively predictable pattern.

A human performs an action.

User
  ↓
Application
  ↓
Database
  ↓
Response

A user opens an application.

They click a button.

The application executes a predetermined operation.

The database changes.

The result appears.

This architecture has powered everything from banking systems to ERP platforms.

But AI agents introduce something different.

Instead of:

Human → Software → Result

we can increasingly build:

Human
  ↓
AI Agent
  ↓
Reason
  ↓
Plan
  ↓
Tools
  ↓
Systems
  ↓
Actions
  ↓
Result

The software isn't simply responding.

It is deciding which operations need to happen.

That is a major architectural shift.

The Difference Between a Chatbot and an Agent

Consider a customer asking:

"Why hasn't my order arrived?"

A chatbot might answer:

"Your order is currently in transit."

An agent could potentially:

Identify the customer.
Query the order database.
Retrieve the shipping record.
Query the logistics provider.
Determine whether the package is delayed.
Check company policy.
Create a support ticket.
Notify the customer.
Escalate the issue if necessary.

The difference is not intelligence alone.

It is agency.

The agent has access to tools.

A Simple Agent Architecture

                    User Request
                         │
                         ▼
                  ┌─────────────┐
                  │ AI Agent    │
                  │             │
                  │ Reason      │
                  │ Plan        │
                  │ Decide      │
                  └──────┬──────┘
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
          Database      API        Search
             │           │           │
             └───────────┼───────────┘
                         ▼
                       Action
                         │
                         ▼
                       Result

This looks simple.

It isn't.

The moment an AI system can take actions, we have to answer questions traditional software didn't have to answer in quite the same way.

What Is the Agent Allowed to Do?

Imagine an AI employee with access to:

Customer database
Payment system
Email
CRM
Internal documents
Cloud infrastructure
Source code

If the model makes a mistake, the consequences aren't just a bad sentence.

It could:

Delete data
Send an email
Approve a payment
Modify infrastructure
Expose private information

This is why agentic AI creates a new security problem.

Recent enterprise security activity reflects this concern. Reuters reported in August 2026 that Obsidian Security raised funding at a $1.1 billion valuation amid demand for security around AI agents accessing sensitive enterprise data; the company said nearly 70% of its clients allowed AI agents to interact with business data.

The interesting thing isn't the funding.

It's the architectural implication.

AI agents need permissions.

Building a Permissioned Agent

Instead of giving an agent unrestricted access:

agent = Agent(
    database="admin",
    email="full",
    payments="full"
)

we should define capabilities.

permissions = {
    "read_orders": True,
    "read_customer": True,
    "send_support_email": True,
    "refund_payment": False,
    "delete_customer": False,
}

Then tools become controlled interfaces.

def get_order(order_id):
    if not permissions["read_orders"]:
        raise PermissionError()

    return database.get_order(order_id)

This resembles traditional security engineering.

The difference is that the caller is now probabilistic.

That makes least privilege even more important.

Agents Need Deterministic Boundaries

I wouldn't want an AI agent deciding everything.

I'd give it deterministic boundaries.

                    AI
                     │
              "I want to refund"
                     │
                     ▼
              Policy Engine
                     │
           ┌─────────┴─────────┐
           ▼                   ▼
       Allowed               Denied
           │
           ▼
        Payment API

The model proposes.

The policy system decides.

This separation is going to become extremely important.

Agentic Systems Will Change Backend Architecture

Traditional backend:

HTTP Request
      ↓
Controller
      ↓
Service
      ↓
Database

Agentic backend:

User
 ↓
Agent
 ↓
Planner
 ↓
Tool Selection
 ↓
Policy Engine
 ↓
Services
 ↓
External APIs
 ↓
Verification
 ↓
Response

Now we need new infrastructure for:

Tool registries
Agent identity
Permissions
Execution traces
Memory
Policy enforcement
Human approval
Audit logs
Cost tracking

In other words:

The enterprise backend is acquiring an operating system for AI workers.

The Hidden Problem: Token Economics

There is another problem people don't talk about enough.

Agents can become expensive.

An ordinary API request might require:

1 request
1 database query
1 response

An agent might perform:

Request
 ↓
Reason
 ↓
Tool selection
 ↓
Tool call
 ↓
Observe result
 ↓
Reason again
 ↓
Another tool
 ↓
Reason again
 ↓
Final response

Each reasoning cycle can consume model inference.

Recent analysis has highlighted this emerging "token tax": agentic systems can consume large numbers of tokens internally for planning, tool selection and iterative reasoning even when the user-facing output is short.

So enterprises will increasingly optimize:

AI Quality
+
Latency
+
Token Usage
+
Tool Calls
+
Infrastructure Cost

This is a new kind of systems engineering.

2. AI-Native Infrastructure: The Computer Is Changing Again

The second quiet transformation is happening below the application layer.

AI has changed what we expect infrastructure to do.

Traditional cloud applications were mostly CPU-oriented.

Application
    ↓
CPU
    ↓
Memory
    ↓
Storage

Modern AI systems increasingly require:

Application
    ↓
Model
    ↓
GPU / Accelerator
    ↓
High-Speed Memory
    ↓
High-Speed Network
    ↓
Storage

This changes the economics and architecture of enterprise computing.

Gartner identifies AI supercomputing platforms and AI-native development platforms among its major 2026 strategic trends.

The New Enterprise Compute Stack

A simplified AI infrastructure stack looks like:

┌──────────────────────────────┐
│ Enterprise Applications      │
├──────────────────────────────┤
│ AI Agents / Models           │
├──────────────────────────────┤
│ Inference Runtime            │
├──────────────────────────────┤
│ GPU / AI Accelerators        │
├──────────────────────────────┤
│ High-Speed Networking        │
├──────────────────────────────┤
│ Storage                      │
├──────────────────────────────┤
│ Compute / Virtualization     │
└──────────────────────────────┘

The important shift is that AI is becoming an infrastructure concern.

It isn't just an application feature.

Why This Matters to Software Engineers

Imagine you're building an application that processes millions of documents.

A traditional architecture might look like:

API
 ↓
CPU workers
 ↓
Database

An AI architecture might look like:

API
 ↓
Queue
 ↓
GPU Workers
 ↓
Model Inference
 ↓
Vector Database
 ↓
Results

Now your architecture needs to understand:

GPU utilization
Model loading
Batch inference
Memory bandwidth
Quantization
Model caching
Inference latency

This is a very different world.

AI-Native Development

There is also a change happening in how software itself is produced.

AI-native development isn't simply:

Developer + Copilot

It increasingly looks like:

Developer
   ↓
AI coding system
   ↓
Planning
   ↓
Code generation
   ↓
Tests
   ↓
Static analysis
   ↓
Review
   ↓
Deployment

The developer increasingly becomes an architect and verifier.

The machine generates more of the implementation.

A Simple AI Development Pipeline

                  Requirement
                       │
                       ▼
                 AI Planner
                       │
                       ▼
                 Code Generator
                       │
                       ▼
                    Tests
                       │
                       ▼
                Static Analysis
                       │
                       ▼
                 Human Review
                       │
                       ▼
                   Deployment

This changes what "productivity" means.

The bottleneck may move from writing code to:

Architecture
Specification
Testing
Verification
Security

That's why I think strong software engineering fundamentals become more valuable, not less.

AI Doesn't Remove Architecture

If anything, it makes architecture more important.

AI can generate:

def create_user():
    ...

But it doesn't automatically know:

Whether the API should be synchronous.
Whether the database needs transactions.
Whether the operation should be idempotent.
Whether a queue is required.
Whether the endpoint should be public.
Whether the data is sensitive.

The hard part of engineering has never been typing.

It has been deciding what should exist.

3. Confidential Computing: Trusting the Machine Less

This is one of the technologies I find particularly interesting.

For decades, security focused heavily on:

Data at rest
Data in transit

We encrypt databases.

We use TLS.

We encrypt backups.

But what happens while data is being processed?

Encrypted Data
     ↓
Application
     ↓
CPU Memory
     ↓
Processing

At some point, the data has to become usable.

Confidential computing attempts to protect workloads and data while they are in use using hardware-backed trusted execution environments.

The Confidential Computing Consortium describes trusted execution environments as a way to isolate memory so that even privileged infrastructure layers such as the host OS or hypervisor can be outside the trust boundary.

This becomes particularly interesting for AI.

Why AI Makes Confidential Computing More Important

Imagine a hospital wants to use an AI model on sensitive medical data.

Traditional architecture:

Hospital Data
      ↓
Cloud
      ↓
AI Model
      ↓
Results

The organization has to trust the infrastructure handling that data.

With confidential computing:

             Cloud Infrastructure
                     │
          ┌──────────┴──────────┐
          │ Trusted Environment │
          │                     │
          │ Data                │
          │ Model               │
          │ Computation         │
          └─────────────────────┘

The goal is to reduce the amount of infrastructure that must be trusted.

Attestation

One of the powerful ideas here is remote attestation.

Instead of simply saying:

"Trust me, I'm running the correct software."

a system can provide cryptographic evidence about the environment it is running in.

Conceptually:

Workload
   ↓
Trusted Environment
   ↓
Measurement
   ↓
Cryptographic Attestation
   ↓
Verifier

Then a service can make decisions based on the attested environment.

This is a fundamentally different trust model.

A Simple Conceptual Attestation Flow

def authorize_workload(attestation):

    if not verify_signature(attestation):
        return False

    if attestation.measurement not in
       APPROVED_MEASUREMENTS:
        return False

    return True

Obviously, production attestation is much more complex.

But the principle is simple:

Don't trust the environment merely because it says it is trustworthy. Verify it.

Confidential Computing + AI Agents

Now combine the first trend with the third.

An AI agent can:

Read sensitive data
Reason over it
Call tools
Make decisions

This creates a massive trust problem.

Confidential computing can become part of the security architecture:

                 AI Agent
                    │
                    ▼
             Policy Layer
                    │
                    ▼
           Confidential Runtime
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
       Secrets              Data
          │                   │
          └─────────┬─────────┘
                    ▼
                Model

This doesn't magically make AI secure.

But it can strengthen the hardware foundation beneath the system.

Research in 2026 has specifically explored confidential computing as a security substrate for agentic AI because autonomous agents can hold credentials, process sensitive context and interact with multiple systems.

4. Edge AI: Intelligence Is Moving Toward the Data

The cloud centralized computing.

The edge is decentralizing it again.

For many years, the architecture looked like:

Device
  ↓
Internet
  ↓
Cloud
  ↓
Compute

But latency-sensitive applications increasingly look like:

Device
  ↓
Edge Compute
  ↓
Cloud

This is especially important for AI.

Why Send Everything to the Cloud?

Imagine a factory camera generating:

30 frames / second

Across:

500 cameras

That's an enormous amount of data.

Sending every raw frame to a distant cloud isn't necessarily efficient.

Instead:

Camera
 ↓
Edge AI
 ↓
"Anomaly detected"
 ↓
Cloud

The edge device processes the raw information.

The cloud receives the important event.

Edge Architecture

              CLOUD
                │
        Global Coordination
                │
       ┌────────┼────────┐
       │        │        │
       ▼        ▼        ▼
     Edge A   Edge B   Edge C
       │        │        │
      IoT      IoT      IoT
    Devices   Devices   Devices

This changes the architecture of enterprise software.

Instead of one centralized application, we get a distributed computing continuum.

Edge AI Implementation

Suppose we have a camera and an object-detection model.

Conceptually:

while True:

    frame = camera.read()

    prediction = model.predict(frame)

    if prediction.confidence > 0.9:

        send_event({
            "type": "anomaly",
            "confidence":
                prediction.confidence
        })

The important design choice is:

The raw frame doesn't necessarily need to leave the edge device.

Only the useful information does.

The Benefits

Edge AI can provide:

Lower latency

Device → Edge

is usually faster than:

Device → Internet → Cloud

Lower bandwidth

Send:

"Anomaly detected"

instead of thousands of images.

Better resilience

The edge system can continue operating even when cloud connectivity disappears.

Privacy

Sensitive data can sometimes remain local.

The Problem With Edge

Edge computing creates a new operational nightmare.

Instead of:

10 servers

you might have:

10,000 edge devices

Now you need:

Remote updates
Device identity
Monitoring
Secure boot
Model deployment
Hardware health monitoring
Offline operation
Configuration management

This is why edge computing isn't simply "put AI on a Raspberry Pi."

It's distributed systems at physical scale.

Edge Model Deployment

A simple model lifecycle might look like:

Model v1
   ↓
Train
   ↓
Validate
   ↓
Package
   ↓
Sign
   ↓
Deploy to 1%
   ↓
Monitor
   ↓
Deploy to 10%
   ↓
Deploy to 100%

This is essentially CI/CD for intelligence.

Edge + Cloud

The strongest architecture probably isn't:

Edge OR Cloud

It's:

Edge
 +
Regional Compute
 +
Cloud

For example:

Sensor
  ↓
Edge inference
  ↓
Regional aggregation
  ↓
Cloud analytics
  ↓
Global model training
  ↓
Updated model
  ↓
Edge deployment

This creates a feedback loop.

5. Domain-Specific AI: Smaller Models, Better Systems

The fifth trend I think deserves more attention is specialization.

The AI conversation has focused heavily on giant general-purpose models.

But enterprises often don't need an AI that knows everything.

They need an AI that understands:

Insurance
Banking
Agriculture
Manufacturing
Law
Healthcare
Logistics
Telecommunications

This is where domain-specific AI becomes interesting.

Gartner lists domain-specific language models among its major 2026 strategic technology trends.

General AI vs Domain AI

Imagine a bank wants a model for:

Loan underwriting

A general model may know about:

History
Science
Programming
Sports
Movies
Politics

But the bank primarily needs:

Credit policies
Risk models
Regulations
Internal procedures
Financial terminology
Customer data

A specialized system can focus on those domains.

Domain AI Architecture

                 General Model
                       │
                       ▼
                Domain Adaptation
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Banking       Legal       Healthcare
        Model         Model          Model
          │            │             │
          ▼            ▼             ▼
      Enterprise    Enterprise    Enterprise
        Data          Data          Data

This doesn't necessarily mean training an enormous model from scratch.

There are many approaches:

Fine-tuning
Retrieval-augmented generation
Distillation
Smaller specialist models
Tool use
Domain-specific embeddings
Structured knowledge bases

RAG: Giving Models Enterprise Memory

Suppose an employee asks:

"What is our company's refund policy?"

Instead of training the model on the policy, we can retrieve the relevant document.

Question
   ↓
Embedding
   ↓
Vector Search
   ↓
Relevant Documents
   ↓
LLM
   ↓
Answer

This is Retrieval-Augmented Generation.

A Minimal RAG Pipeline

def answer(question):

    query_vector = embed(question)

    documents = vector_db.search(
        query_vector,
        top_k=5
    )

    context = "\n".join(
        doc.text for doc in documents
    )

    prompt = f"""
    Answer using only this context:

    {context}

    Question:
    {question}
    """

    return model.generate(prompt)

This is a deceptively simple pattern.

But it changes enterprise software dramatically.

The model doesn't need to memorize the company.

It needs access to the company's knowledge.

Enterprise AI Is Becoming a Data Architecture Problem

This is one of the most important lessons.

Companies may buy the same model.

But their internal data is different.

Therefore the competitive advantage often becomes:

Model
+
Data
+
Workflows
+
Tools
+
Distribution

not merely:

Model

The Rise of Small Models

A giant model isn't always the best solution.

Suppose the task is:

Classify support tickets

Maybe we need:

Model size: small
Latency: 10 ms
Cost: tiny
Accuracy: 98%

A giant model might provide:

Latency: 500 ms
Cost: high
Accuracy: 98.5%

The extra intelligence may not justify the cost.

Enterprise AI will increasingly become an optimization problem.

Quality
   ×
Latency
   ×
Cost
   ×
Privacy
   ×
Reliability

Putting the Five Trends Together

This is where things become really interesting.

These technologies don't exist independently.

They reinforce one another.

Imagine a future manufacturing company.

It has:

Robots
Sensors
AI agents
Edge inference
Confidential computing
Domain-specific models
Cloud infrastructure

The architecture might look like this:

                         ENTERPRISE
                             │
                     ┌───────┴───────┐
                     │               │
                 AI Agents      Domain Models
                     │               │
                     └───────┬───────┘
                             │
                      Policy Engine
                             │
                    Confidential Runtime
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
            Cloud          Edge          Robotics
              │              │              │
              └──────────────┼──────────────┘
                             │
                           Data
                             │
                        Knowledge

This isn't science fiction.

It's an architectural direction.

The Enterprise Is Becoming a Distributed Intelligence System

Traditional enterprise:

People
  ↓
Applications
  ↓
Databases

Emerging enterprise:

People
   │
   ▼
Agents
   │
   ▼
Applications
   │
   ├──────────► APIs
   │
   ├──────────► Databases
   │
   ├──────────► Models
   │
   ├──────────► Edge Devices
   │
   └──────────► Physical Machines

The application is no longer the only actor.

Machines are beginning to participate in workflows.

What Happens to the Backend Engineer?

I think the role changes.

The backend engineer used to focus heavily on:

APIs
Databases
Queues
Services
Authentication

Those things aren't going away.

But now we increasingly need:

Agent APIs
Model gateways
Tool permissions
Vector databases
Event-driven workflows
Inference infrastructure
AI observability
Policy engines

The backend becomes the connective tissue between intelligence and the real world.

A New Kind of API

Traditional API:

POST /orders

An agent-oriented API might expose capabilities:

create_order
cancel_order
lookup_customer
check_inventory
request_refund

The difference is subtle.

The API is becoming a tool interface for intelligent software.

That means APIs need stronger contracts.

APIs Need to Become More Machine-Friendly

An AI agent needs to understand:

What does this tool do?

What parameters does it accept?

What permissions are required?

What can go wrong?

Is this operation reversible?

Does it cost money?

So API design increasingly becomes partly about machine discoverability.

Observability Changes Too

Traditional monitoring:

CPU
Memory
Latency
Errors
Requests

AI systems add:

Token usage
Model latency
Tool calls
Prompt failures
Agent loops
Hallucinations
Policy violations
Model versions
Retrieval quality

Imagine an agent making 17 tool calls for something that should require 3.

Your infrastructure might be healthy.

Your software might still be terrible.

Agent Tracing

We might need traces like:

Request #83921

Agent
 │
 ├── Model call: 320ms
 │
 ├── Search: 40ms
 │
 ├── Database: 12ms
 │
 ├── Model call: 510ms
 │
 ├── Payment API: 210ms
 │
 └── Model call: 280ms

Total: 1.37s
Tokens: 14,280
Tools: 3

This becomes the equivalent of distributed tracing for intelligent systems.

The New Reliability Problem

Traditional software is usually deterministic.

Input A
 ↓
Output B

AI is probabilistic.

Input A
 ↓
Possible Output B
Possible Output C
Possible Output D

This changes testing.

Instead of only:

assert result == expected

we may need:

assert passes_policy(result)

assert contains_required_information(result)

assert does_not_expose_sensitive_data(result)

The test surface becomes different.

Enterprises Will Need AI Governance as Engineering

Governance can't just be a PDF.

It needs to become infrastructure.

Imagine:

                  AI Request
                       │
                       ▼
                 Policy Engine
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
    Identity        Data Rules      Action Rules
        │              │              │
        └──────────────┼──────────────┘
                       ▼
                    Decision

Policy should be executable.

For example:

if agent.role == "support":
    deny("delete_customer")

if data.classification == "restricted":
    require_human_approval()

if action == "refund" and amount > 1000:
    require_human_approval()

Now governance becomes software.

That is a much more powerful model.

What Companies Should Actually Do

I wouldn't recommend that every enterprise immediately deploy 500 AI agents.

Technology adoption should follow the problem.

I'd start with workflows.

Find processes that are:

Repetitive
Data-heavy
Well-defined
Measurable
Low-risk

Then introduce AI.

Step 1: Find the Workflow

Example:

Customer Support Ticket

Current process:

Ticket
 ↓
Human reads
 ↓
Find customer
 ↓
Find order
 ↓
Find policy
 ↓
Write response

Potential AI-assisted process:

Ticket
 ↓
Agent
 ├── Customer lookup
 ├── Order lookup
 ├── Policy retrieval
 └── Draft response
        ↓
     Human review

This is a much safer starting point than:

"Let's let an AI run customer support."

Step 2: Introduce Deterministic Tools

Give the agent tools.

tools = [
    get_customer,
    get_order,
    search_policy,
    draft_response,
]

Don't initially give it:

delete_database

The principle is:

Start with read-only capabilities.

Then gradually increase autonomy.

Step 3: Add Observability

Track:

Agent requests
Tool calls
Failures
Latency
Cost
Human overrides
Policy violations

You need data before you can optimize.

Step 4: Add Permissions

Every agent should have an identity.

Agent:
    support-agent-01

Permissions:
    customer.read
    order.read
    policy.read
    response.draft

Not:

admin.*

Step 5: Measure Business Outcomes

Don't measure:

Number of AI calls

Measure:

Resolution time
Cost per ticket
Customer satisfaction
Human review time
Error rate
Escalation rate

Technology isn't the outcome.

The business result is.

The Architecture I Would Build

If I were designing a modern enterprise platform around these trends, I'd think about it as layers.

┌──────────────────────────────────────┐
│          Business Applications       │
├──────────────────────────────────────┤
│       Agents / AI Workflows          │
├──────────────────────────────────────┤
│      Domain Models / RAG             │
├──────────────────────────────────────┤
│       Policy + Identity              │
├──────────────────────────────────────┤
│       Observability                  │
├──────────────────────────────────────┤
│      Confidential Runtime            │
├──────────────────────────────────────┤
│      Cloud + Edge Compute            │
├──────────────────────────────────────┤
│      Storage + Networking            │
└──────────────────────────────────────┘

This is very different from the traditional:

Frontend
 ↓
Backend
 ↓
Database

architecture.

The Biggest Shift Isn't AI

This might sound strange considering the article is about emerging technology.

But I don't think AI itself is the deepest change.

The deeper change is autonomy.

Software is gradually moving from:

Passive tools

to:

Active participants

A database waits for a query.

An API waits for a request.

A traditional application waits for a user.

An agent can initiate work.

That changes everything.

Software Is Becoming More Like an Organization

Think about a company.

There are:

Employees
Managers
Policies
Departments
Systems
Communication
Permissions

Now imagine representing some of those functions in software.

Sales Agent
Support Agent
Research Agent
Finance Agent
Engineering Agent
Security Agent

They communicate through APIs.

             Orchestrator
                  │
      ┌───────────┼───────────┐
      ▼           ▼           ▼
    Sales      Support     Finance
    Agent       Agent       Agent
      │           │           │
      └───────────┼───────────┘
                  ▼
              Enterprise
               Systems

This is why multiagent architectures are receiving attention: the system can divide complex workflows into specialized capabilities rather than relying on one monolithic agent. Gartner explicitly lists multiagent systems among its 2026 strategic trends.

But We Should Be Careful

There is a temptation to assume:

More agents = better

Not necessarily.

Every agent adds:

Latency
Cost
Failure modes
Security boundaries
Debugging complexity

A distributed system with 50 agents can become a distributed systems nightmare.

Sometimes one deterministic function is better than an agent.

For example:

calculate_tax(order)

doesn't need an LLM.

Use deterministic software where deterministic software works.

Use intelligence where ambiguity exists.

That distinction will become extremely valuable.

The Future Enterprise Will Be Hybrid

I don't think the future looks like:

Everything = AI

It looks like:

Deterministic Software
+
AI
+
Humans
+
Robotics
+
Edge Systems

Each should handle what it is good at.

                    Problem
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
   Deterministic       AI            Human
      Logic          Reasoning       Judgment
        │              │              │
        └──────────────┼──────────────┘
                       ▼
                     Result

The best systems will orchestrate all three.

What This Means for Startups

There is a huge opportunity here.

You don't necessarily need to build another generic AI chatbot.

You can build infrastructure around the emerging architecture.

For example:

Agent security

Agent
 ↓
Identity
 ↓
Permissions
 ↓
Policy
 ↓
Audit

Edge AI infrastructure

Model
 ↓
Deployment
 ↓
Device
 ↓
Monitoring
 ↓
Updates

Domain AI

Enterprise Data
 ↓
Retrieval
 ↓
Specialized Model
 ↓
Workflow

AI observability

Agent
 ↓
Trace
 ↓
Cost
 ↓
Latency
 ↓
Quality

These are infrastructure problems.

And infrastructure problems can become enormous companies.

What This Means for African Technology

I also think these trends create an interesting opportunity for African engineers.

There is a tendency to think technological innovation requires recreating Silicon Valley.

I don't think it does.

Many African markets have problems that are particularly suitable for these architectures.

Agriculture.

Logistics.

Healthcare.

Banking.

Energy.

Telecommunications.

Mining.

Education.

Consider agriculture.

Farm Sensors
     ↓
Edge AI
     ↓
Local Analysis
     ↓
Agricultural Agent
     ↓
Market Data
     ↓
Farmer Recommendation

Or logistics:

Vehicle
 ↓
Edge Device
 ↓
Route Intelligence
 ↓
Cloud
 ↓
Optimization Agent

Or healthcare:

Local Device
 ↓
Edge Inference
 ↓
Confidential Processing
 ↓
Clinical Workflow
 ↓
Human Decision

The important thing is not copying someone else's application.

It is understanding the underlying architecture and applying it to local problems.

The Engineering Principles Behind All Five Trends

After looking at these technologies, I think several engineering principles emerge.

1. Move intelligence closer to the decision

Sometimes that means:

Cloud

Sometimes:

Edge

Sometimes:

Human

The question is:

Where should the decision happen?

2. Minimize trust

Don't assume:

The cloud is trusted.
The agent is trusted.
The model is trusted.
The network is trusted.

Instead:

Identity
+
Policy
+
Attestation
+
Verification

3. Make capabilities explicit

An agent shouldn't have:

everything()

It should have:

read_customer()
read_order()
create_ticket()

Explicit interfaces create safer systems.

4. Keep deterministic systems deterministic

Don't use an LLM for:

Tax calculation
Balance calculation
Permission checking
Currency conversion

Use traditional software.

AI should handle ambiguity.

5. Design for observability

Every intelligent action should ideally be traceable.

Who?
What?
Why?
Which model?
Which data?
Which tools?
What happened?
How much did it cost?

The Enterprise Architecture of Tomorrow

Put everything together and the picture becomes fascinating.

                              USERS
                                │
                                ▼
                         Experience Layer
                                │
                                ▼
                     Intelligent Orchestration
                                │
              ┌─────────────────┼─────────────────┐
              ▼                 ▼                 ▼
           Agents          Applications       Humans
              │                 │                 │
              └─────────────────┼─────────────────┘
                                ▼
                         Policy + Identity
                                │
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
             Models           APIs            Data
                │               │               │
                └───────────────┼───────────────┘
                                ▼
                       Confidential Runtime
                                │
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
              Cloud           Edge          Physical AI
                │               │               │
                └───────────────┼───────────────┘
                                ▼
                          Infrastructure

That is the direction I see.

Not one giant AI.

Not one giant cloud.

A distributed ecosystem of intelligence.

Final Thoughts

The most important technology trends aren't always the ones dominating social media.

Sometimes the real transformation is happening in architecture diagrams.

I think these five trends deserve serious attention:

Agentic AI is turning software from a passive tool into an active participant.

AI-native infrastructure is changing what enterprise computing looks like underneath applications.

Confidential computing is creating stronger trust boundaries for sensitive workloads.

Edge AI and physical intelligence are moving computation toward the places where decisions actually happen.

Domain-specific AI is pushing enterprises away from the idea that one enormous general-purpose model is the answer to everything.

And together, they create something bigger.

They create a new kind of enterprise.

One where:

Software can reason.
Software can act.
Machines can perceive.
Data can remain closer to its source.
Infrastructure can verify itself.
Models can specialize.

But I don't think the winning companies will simply be the ones with the most AI.

They'll be the ones that understand systems.

Because once AI becomes capable of acting, the difficult problems stop being:

"Can the model generate an answer?"

The difficult questions become:

Should it act?

What is it allowed to access?

Where should computation happen?

Can we verify the environment?

Can we explain what happened?

Can we afford the inference?

What happens when the model is wrong?

What happens when the network disappears?

What happens when an agent behaves unexpectedly?

These are engineering questions.

And that's why I find this period so interesting.

We're not merely adding intelligence to existing software.

We're redesigning the machinery around it.

The enterprise application of the past looked something like:

Human
  ↓
Application
  ↓
Database

The emerging enterprise looks more like:

                         HUMAN
                           │
                           ▼
                    INTELLIGENT SYSTEM
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
           AGENTS        MODELS        HUMANS
             │             │             │
             └─────────────┼─────────────┘
                           ▼
                    POLICY + IDENTITY
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
            CLOUD         EDGE       PHYSICAL
              │            │            │
              └────────────┼────────────┘
                           ▼
                         DATA

And that is the real story.

The future of enterprise technology isn't simply more powerful software.

It's software that can perceive, reason, coordinate, verify, and act across an increasingly distributed world.

The companies that understand that architecture early will have an enormous advantage.

Because the next generation of enterprise systems won't just execute instructions.

They'll participate in the work.

Building Your Own SQL Database

Derek mwale — Tue, 04 Aug 2026 00:08:50 +0000

From Bytes on Disk to Tables You Can Query

Every developer uses databases.

Whether you're building a small blog, a banking platform, an e-commerce store, or a distributed cloud service, sooner or later your application needs somewhere to store information.

We write SQL queries every day.

SELECT * FROM users;

INSERT INTO products (...);

UPDATE orders SET ...;

Most of the time, we stop there.

We trust PostgreSQL, MySQL, SQLite, or SQL Server to do the rest.

But have you ever wondered what actually happens after you press Enter?

How does a database know where your data lives?

How does it find rows without reading the entire file?

How does it survive power failures?

How does it store millions—even billions—of records efficiently?

When I first started backend development, I imagined databases as giant spreadsheets.

As I learned more, I realized they are closer to miniature operating systems.

They manage memory.

They schedule disk access.

They recover from crashes.

They optimize execution plans.

They maintain consistency while thousands of users access data simultaneously.

Building a database completely changed the way I think about software.

In this series, we'll build the foundations of our own SQL database in Rust.

Not because we're trying to replace PostgreSQL.

But because building one is one of the best ways to understand how modern software stores, retrieves, and protects information.

Let's begin with the most fundamental question.

Where does a database actually keep your data?

A Database Starts as a File

Many beginners imagine a database as something abstract living inside a server.

Reality is much simpler.

Every database ultimately stores information in files.

Application

      │

      ▼

 SQL Query

      │

      ▼

Database Engine

      │

      ▼

Database File

Whether you're using SQLite or PostgreSQL, the lowest layer eventually writes bytes to storage.

The database is simply responsible for organizing those bytes intelligently.

Thinking Beyond Tables

Applications show data as tables.

Users

+----+--------+

| ID | Name   |

+----+--------+

| 1  | Derek  |

| 2  | Alex   |

+----+--------+

The disk never stores tables like this.

Instead, everything becomes binary.

Tables are abstractions.

The database translates between human-friendly structures and machine-friendly bytes.

Layers of a Database

A modern SQL database resembles a layered architecture.

                SQL

                 │

                 ▼

          Query Engine

                 │

                 ▼

         Storage Engine

                 │

                 ▼

            File System

                 │

                 ▼

             SSD / HDD

Each layer has one responsibility.

That's good software engineering.

Designing Our Database

Our database will be called:

TinyDB

Project structure.

tinydb/

├── storage/

├── table/

├── page/

├── sql/

├── parser/

├── executor/

├── catalog/

├── buffer/

└── main.rs

Notice something interesting.

Large databases are simply well-organized software projects.

Representing a Row

Let's begin with one row.

Rust.

pub struct User {

    pub id: u64,

    pub name: String,

}

Simple.

Applications think in structures.

Storage engines think differently.

Serialization

Before writing data to disk, we must convert structures into bytes.

User

↓

Serialize

↓

Bytes

↓

Disk

The reverse process is called deserialization.

Example Serialization

Suppose we have:

User {

    id: 1,

    name: "Derek"

}

Serialized representation.

01 00 00 00

05

44 65 72 65 6B

The exact format is our choice.

That's one of the fascinating parts of building storage engines.

Storage Pages

Databases rarely read individual rows.

Instead they organize storage into pages.

Typically:

4096 Bytes

↓

One Page

Every read.

Every write.

Occurs page by page.

Why Pages?

Imagine a million users.

Reading one row at a time would be extremely inefficient.

Instead:

Disk

┌───────────┐

│ Page 1    │

├───────────┤

│ Page 2    │

├───────────┤

│ Page 3    │

└───────────┘

Pages minimize disk operations.

Designing a Page

Rust.

pub struct Page {

    pub id: u32,

    pub data: Vec<u8>,

}

Initially...

Just raw bytes.

Higher layers interpret them.

Table Storage

Suppose each page stores several rows.

Page

+-------------------------+

Row

Row

Row

Row

+-------------------------+

Eventually pages fill up.

The database allocates another.

Building the Storage Engine

Architecture.

Application

      │

      ▼

Table

      │

      ▼

Page Manager

      │

      ▼

File Manager

      │

      ▼

Disk

Each component solves one problem.

Writing Pages

Rust implementation.

use std::fs::File;

use std::io::Write;

fn write_page(

    file: &mut File,

    page: &Page,

) {

    file.write_all(

        &page.data

    ).unwrap();

}

Simple.

Yet this is the foundation of every database.

Reading Pages

Reading becomes equally straightforward.

use std::io::Read;

fn read_page(

    file: &mut File,

    page: &mut Page,

) {

    file.read_exact(

        &mut page.data

    ).unwrap();

}

Storage engines spend much of their lives performing operations like these.

Building a Table

Now we combine pages into tables.

Rust.

pub struct Table {

    pub name: String,

    pub pages:

        Vec<Page>,

}

Eventually tables may contain thousands of pages.

Or millions.

Inserting Data

Suppose we execute:

INSERT INTO users

VALUES (1,'Derek');

Flow.

SQL

↓

Insert Command

↓

Serialize Row

↓

Find Free Page

↓

Write Bytes

↓

Commit

The SQL statement eventually becomes disk writes.

Reading Data

Now imagine:

SELECT *

FROM users;

Execution.

Read Page

↓

Deserialize Rows

↓

Return Results

Queries simply reverse the insertion process.

Catalog

How does the database know which tables exist?

Using a catalog.

Catalog

+-----------+

Users

Orders

Products

Invoices

+-----------+

Think of it as metadata describing the database itself.

Catalog Structure

Rust.

pub struct Catalog {

    tables:

        Vec<Table>,

}

Later this grows considerably.

Indexes.

Schemas.

Constraints.

Views.

Statistics.

Everything begins here.

Buffer Pool

Reading directly from disk every time would be painfully slow.

Instead databases cache pages.

Disk

↓

Buffer Pool

↓

Application

Recently used pages remain in memory.

Why Buffer Pools Matter

Suppose ten users request the same page.

Without caching.

Disk

↓

Disk

↓

Disk

↓

Disk

With caching.

Disk

↓

Memory

↓

Memory

↓

Memory

Much faster.

Simplified Buffer Manager

Rust.

use std::collections::HashMap;

pub struct BufferPool {

    pages:

    HashMap<u32, Page>,

}

Whenever possible:

Read from memory.

Otherwise:

Read from disk.

Database Flow

Putting everything together.

Application

↓

SQL

↓

Table Manager

↓

Buffer Pool

↓

Storage Engine

↓

Disk

Notice how SQL never interacts with files directly.

Every layer abstracts the one below it.

Complete Architecture

                 SQL Client

                      │

                      ▼

                Query Layer

                      │

                      ▼

              Table Manager

                      │

                      ▼

               Buffer Manager

                      │

         ┌────────────┼─────────────┐

         ▼            ▼             ▼

      Page 1       Page 2        Page 3

         │            │             │

         └────────────┼─────────────┘

                      ▼

                Database File

                      │

                      ▼

                   SSD / HDD

Even though our database is still small, the architecture already resembles production systems.

Implementation Summary

At this point, TinyDB supports the core storage concepts found in real databases:

Rows represented as Rust structures.
Serialization of rows into bytes.
Fixed-size storage pages.
Tables composed of multiple pages.
A storage engine that reads and writes pages to disk.
A simple catalog to track tables.
A buffer pool that caches frequently accessed pages.

These pieces may appear simple, but they form the foundation of nearly every relational database.

Looking Ahead

So far, we've focused entirely on how data is stored.

But a database becomes truly useful when it understands SQL.

In Part 2, we'll build our own SQL engine from scratch.

We'll write a lexer to break SQL into tokens, build a parser that constructs an Abstract Syntax Tree (AST), implement a query planner, and create an execution engine capable of processing commands like:

CREATE TABLE users (...);

INSERT INTO users VALUES (...);

SELECT * FROM users WHERE id = 1;

We'll explore how databases transform human-readable SQL into executable operations, introduce table scans, filtering, and projections, and compare our design to the internal architecture of PostgreSQL and SQLite.

By the end of the next article, TinyDB won't just store data—it will understand and execute SQL.

Final Thoughts

One of the biggest surprises in building a database is realizing how much of it has nothing to do with SQL.

Before a parser, optimizer, or query planner can exist, the database must solve a far more fundamental problem: how to organize information on disk efficiently and reliably.

Everything starts with bytes.

Rows become serialized records.

Records are packed into pages.

Pages are written to files.

Files become tables.

Tables become databases.

Only then do higher-level abstractions like SQL begin to make sense.

That progression mirrors software engineering as a whole.

The most sophisticated systems are often built from remarkably simple layers, each solving one well-defined problem.

Understanding those layers gives you a deeper appreciation for every query you write.

The next time you execute SELECT * FROM users, you'll know that beneath those four words lies an intricate system of pages, buffers, storage engines, and carefully organized bytes working together to make data appear effortless.

And we're only just getting started.

Building Your Own SQL Database (Part 2)

Building a SQL Parser, Query Planner, and Execution Engine

In Part 1, we built the foundation of our database.

We learned that databases are not magical systems. At their core, they organize bytes on disk into pages, pages into tables, and tables into a storage engine capable of persisting information.

By the end of Part 1, our TinyDB architecture looked like this:

Application
      │
      ▼
 Table Manager
      │
      ▼
 Buffer Pool
      │
      ▼
 Storage Engine
      │
      ▼
 Database File

Our database could store information.

But there was one major problem.

Nobody wants to interact with raw pages or serialized bytes.

Developers speak SQL.

SELECT *
FROM users
WHERE id = 10;

The computer does not.

Computers understand instructions.

Somewhere between SQL and the storage engine, the database must translate human language into executable operations.

That's exactly what we're building today.

By the end of this article, TinyDB will understand SQL, build an Abstract Syntax Tree (AST), generate a query plan, and execute queries against our storage engine.

This is where our database begins to feel alive.

SQL Is Just Another Language

We often think of SQL as something special.

It isn't.

It's simply a programming language.

Like Rust.

Like Python.

Like JavaScript.

Every programming language follows the same pipeline.

Source Code

↓

Lexer

↓

Tokens

↓

Parser

↓

AST

↓

Execution

Databases follow exactly the same idea.

The Journey of a Query

Suppose the user writes:

SELECT name
FROM users
WHERE id = 5;

Internally our database performs:

SQL

↓

Lexer

↓

Tokens

↓

Parser

↓

AST

↓

Query Planner

↓

Execution Engine

↓

Storage Engine

↓

Results

Notice how SQL disappears very early.

Everything afterward operates on structured objects.

Step 1 — Lexical Analysis

The lexer breaks raw text into meaningful pieces.

Input:

SELECT name
FROM users;

Output:

SELECT

IDENTIFIER(name)

FROM

IDENTIFIER(users)

SEMICOLON

These pieces are called tokens.

Designing Tokens

Rust.

pub enum Token {

    Select,

    Insert,

    Update,

    Delete,

    From,

    Where,

    Identifier(String),

    Number(u64),

    String(String),

    Comma,

    Semicolon,

}

Every SQL statement becomes a stream of these objects.

Example Lexer

Suppose our input is:

SELECT id FROM users;

The lexer produces:

+----------------+

SELECT

Identifier(id)

FROM

Identifier(users)

Semicolon

+----------------+

Simple.

Deterministic.

Fast.

Why Tokenization Matters

Imagine parsing SQL character by character.

S

E

L

E

C

T

Very difficult.

Instead we work with meaningful units.

SELECT

Much easier.

Parser

Now the parser transforms tokens into meaning.

Example.

SELECT

Identifier(name)

FROM

Identifier(users)

Becomes:

Select Query

├── Columns

│      name

└── Table

       users

This structure is called an Abstract Syntax Tree.

Building the AST

Rust.

pub struct SelectStatement {

    pub columns:

        Vec<String>,

    pub table: String,

}

Notice something important.

The SQL text disappears.

Only structure remains.

AST Visualization

             SELECT

             /     \

      Columns      Table

        │            │

      name         users

This tree represents the query.

Every SQL database builds something similar.

Parsing SELECT

Pseudo-code.

parse_select()

↓

expect SELECT

↓

parse columns

↓

expect FROM

↓

parse table

↓

return AST

Every parser follows a predictable sequence.

Supporting WHERE

Now let's extend our AST.

SELECT *

FROM users

WHERE id = 7;

Rust.

pub struct Condition {

    pub column: String,

    pub value: Value,

}

Updated statement.

pub struct SelectStatement {

    pub columns:

        Vec<String>,

    pub table: String,

    pub condition:

        Option<Condition>,

}

Our parser now understands filtering.

AST with WHERE

SELECT

├── Columns

│

├── Table

│

└── WHERE

       │

     id = 7

The AST captures meaning—not syntax.

Query Planning

Parsing tells us what the user wants.

Planning determines how to retrieve it.

Example.

SELECT *

FROM users;

Planner chooses:

Full Table Scan

Different query.

SELECT *

FROM users

WHERE id=5;

Planner may choose:

Index Scan

Different execution.

Same SQL language.

Query Plan

Our first planner is extremely simple.

SELECT

↓

Table Scan

↓

Filter

↓

Projection

↓

Results

Every database starts here.

Representing Plans

Rust.

pub enum Plan {

    TableScan,

    Filter,

    Projection,

}

Later this grows into dozens of node types.

Table Scan

Suppose we have:

Users

1 Derek

2 Alex

3 Sarah

Table scan means:

Read Every Row

Simple.

Not always efficient.

Executing a Scan

Rust.

for row in table.rows() {

    println!("{:?}", row);

}

That's literally a table scan.

Filtering Rows

Now suppose:

WHERE id = 2

Execution becomes:

Read Row

↓

Compare

↓

Keep?

↓

Yes

↓

Return

Rust.

if row.id == 2 {

    result.push(row);

}

Projection

Users often request only specific columns.

SELECT name

Instead of:

ID

Name

Email

We return:

Name

Projection removes unnecessary data.

Execution Pipeline

Putting everything together.

Table

↓

Scan

↓

Filter

↓

Projection

↓

Results

Modern databases still follow this concept.

Example Execution

Suppose our table contains:

1 Derek

2 Alex

3 Sarah

Query.

SELECT name

FROM users

WHERE id = 2;

Execution.

Read Derek

↓

No

↓

Read Alex

↓

Yes

↓

Return Alex

↓

Read Sarah

↓

No

Supporting INSERT

INSERT follows another execution path.

INSERT INTO users

VALUES (4,'Mary');

Pipeline.

SQL

↓

Lexer

↓

Parser

↓

Insert AST

↓

Serialize Row

↓

Storage Engine

↓

Disk

Insert AST

Rust.

pub struct InsertStatement {

    pub table: String,

    pub values:

        Vec<Value>,

}

Different SQL.

Different AST.

Same architecture.

Supporting CREATE TABLE

Example.

CREATE TABLE users (

id INT,

name TEXT

);

AST.

pub struct CreateTable {

    pub name: String,

    pub columns:

        Vec<Column>,

}

The catalog eventually stores this information.

Complete SQL Pipeline

             SQL

              │

              ▼

            Lexer

              │

              ▼

            Parser

              │

              ▼

             AST

              │

              ▼

       Query Planner

              │

              ▼

     Execution Engine

              │

              ▼

      Storage Engine

              │

              ▼

            Disk

Every layer has a single responsibility.

Why Query Planners Exist

Imagine two million users.

Query:

SELECT *

FROM users

WHERE id = 900;

Planner has choices.

Option 1

Table Scan

2 Million Rows

-----------------

Option 2

Index Lookup

1 Row

Same answer.

Massively different performance.

We'll build indexes in Part 3.

Current Architecture

                 SQL Client

                      │

                      ▼

                   Lexer

                      │

                      ▼

                   Parser

                      │

                      ▼

                    AST

                      │

                      ▼

                Query Planner

                      │

                      ▼

             Execution Engine

                      │

          ┌───────────┼────────────┐

          ▼           ▼            ▼

     Table Scan    Filter     Projection

                      │

                      ▼

               Storage Engine

                      │

                      ▼

                Database File

TinyDB is beginning to resemble a real relational database.

Comparing TinyDB to PostgreSQL

Component	TinyDB	PostgreSQL
Lexer	✅	✅
Parser	✅	✅
AST	✅	✅
Query Planner	Basic	Cost-Based
Execution Engine	Basic	Volcano Executor
Storage Engine	Basic	Advanced
Optimizer	Planned	Sophisticated

Although simplified, the architectural flow is remarkably similar.

What's Coming Next

TinyDB can now understand SQL and execute simple queries.

But performance is still terrible.

Every lookup scans the entire table.

If our database stores ten million users, finding one record means examining ten million rows.

Real databases don't work that way.

In Part 3, we'll solve this problem by building one of the most important data structures in computer science: the B-Tree.

We'll explore clustered and secondary indexes, implement efficient key lookups, introduce a buffer manager for intelligent caching, build a Write-Ahead Log (WAL) to survive crashes, and add transactions with Multi-Version Concurrency Control (MVCC).

By the end of the next article, TinyDB will begin behaving much more like PostgreSQL or MySQL, capable of handling fast queries while maintaining consistency and durability.

Final Thoughts

One of the biggest surprises in building a database is discovering that SQL is only a tiny part of the system.

The language itself is relatively simple.

The real engineering lies in everything that happens after the parser finishes its work.

A database reads text, transforms it into tokens, builds a structured representation, plans an efficient execution strategy, and finally coordinates with the storage engine to retrieve or modify data.

Those stages mirror the design of compilers, interpreters, and even operating systems.

Different domains.

The same architectural thinking.

Understanding this pipeline fundamentally changes how you write SQL.

A query stops being just a string.

It becomes a program that the database must parse, optimize, and execute.

And once you begin thinking that way, concepts like indexes, execution plans, and query optimization become far less mysterious.

Instead of seeing SQL as magic, you start seeing it for what it truly is: another elegant programming language executed by one of the most sophisticated pieces of software ever built.

Building Your Own SQL Database (Part 3)

Indexes, B-Trees, Transactions, and Making TinyDB Feel Like a Real Database

In Part 1, we built the foundation.

We learned that databases are ultimately machines that transform structured information into carefully organized bytes on disk.

We created:

Pages
Tables
Serialization
Storage engines
Buffer pools

In Part 2, we gave our database a language.

TinyDB learned how to understand SQL.

We built:

A lexer
A parser
An Abstract Syntax Tree
A query planner
An execution engine

Our architecture evolved into:

```text id="h5d7mq"
SQL

             │

             ▼

          Parser

             │

             ▼

      Query Planner

             │

             ▼

    Execution Engine

             │

             ▼

    Storage Engine

             │

             ▼

          Disk




But there is a major problem.

Our database is painfully slow.

Why?

Because we are searching like beginners.

---

# The Problem With Full Table Scans

Imagine we have one billion users.

Our table:



```text id="r2q5jz"
Users

1  Derek

2  Alex

3  Sarah

...

1000000000 John

Now execute:

SELECT *

FROM users

WHERE id = 999999999;

Our database currently does this:

```text id="f2m8qy"
Read Row 1

↓

Check

↓

Read Row 2

↓

Check

↓

Read Row 3

↓

Check

...

↓

Read Row 999999999

↓

Found




This is called a **full table scan**.

Complexity:



```plaintext
O(n)

The bigger the database becomes, the slower it gets.

Production databases cannot work this way.

They need indexes.

What Is an Index?

An index is a separate data structure designed to find information faster.

Think about a dictionary.

Without alphabetical ordering:

```text id="7m4z1p"
apple

zebra

computer

banana

network




Finding "computer" requires checking every word.

With an index:



```text id="n8s7vz"
A

B

C  → computer

D

E

The search becomes much faster.

Databases use the same idea.

Database Index Architecture

Without an index:

```text id="f0kqxa"
Query

↓

Table

↓

Scan Every Row

↓

Result




With an index:



```text id="zq2m3h"
Query

 ↓

Index

 ↓

Find Location

 ↓

Read Row

 ↓

Result

The index acts like a roadmap.

Choosing the Right Data Structure

Many structures can create indexes:

Hash tables
Binary trees
Skip lists
B-Trees
B+ Trees

Most relational databases rely heavily on B-Trees.

Why?

Because databases are not only searching memory.

They are searching disks.

Understanding B-Trees

A normal binary tree looks like:

```text id="7xq4kl"

         50

        /  \

      25    75

     / \    / \

   10 30  60 90




Each node has two children.

A B-Tree expands this idea.

Instead of two children:

A node can have hundreds.



```text id="m2e8k4"

              [50|100|150]

            /      |      \

          /        |       \

     many keys  many keys  many keys

This reduces the height of the tree.

Why B-Trees Are Perfect for Databases

Disk access is expensive.

Imagine:

```text id="u8b3s1"
Memory

↓

Fast

Disk

↓

Slow




A binary tree with one billion records might require many disk reads.

A B-Tree keeps the height small.

Example:



```plaintext
B-Tree height: 3

One lookup:

Root
 ↓
Child
 ↓
Leaf

Only a few disk operations.

Building Our B-Tree

Let's create a simplified version.

```rust id="w8j3fr"
pub struct BTreeNode {

pub keys: Vec<u64>,

pub children:

    Vec<u64>,

pub leaf: bool,

}




Each node stores:

* Keys
* Child pointers
* Whether it is a leaf

---

# Example Index

Suppose we insert:



```plaintext
10
20
30
40
50

Our tree:

```text id="g2a9pm"

          [30]

         /    \

   [10|20]   [40|50]




Searching:



```plaintext
Find 40

↓

Compare with 30

↓

Go right

↓

Find 40

Much faster.

Index Lookup Flow

```text id="j5w0pk"
SELECT *

WHERE id = 50;

    SQL

     │

     ▼

   Planner

     │

     ▼

   Index

     │

     ▼

B-Tree Search

     │

     ▼

Row Location

     │

     ▼

   Table




---

# Adding Indexes to TinyDB

Our table now becomes:



```text id="4q1q5n"
Table

 ┌──────────────┐

 │ Rows         │

 └──────────────┘


 Index

 ┌──────────────┐

 │ B-Tree       │

 └──────────────┘

The index does not replace data.

It points to data.

Storing Row Locations

Instead of storing complete rows:

```text id="j8g6mb"
50 → Derek




We store:



```text id="c7v1zq"
50 → Page 15 Offset 200

Then:

Find location

↓

Read page

↓

Return row

Implementing Index Entries

Rust:

```rust id="w4d0kr"
pub struct IndexEntry {

pub key: u64,

pub page_id: u32,

pub offset: usize,

}




The index becomes a navigation system.

---

# The Need for Transactions

Indexes solve searching.

But databases have another massive problem:

Multiple users changing data at the same time.

Imagine:

User A:



```sql
UPDATE accounts

SET balance=500;

User B:

UPDATE accounts

SET balance=700;

Who wins?

The Lost Update Problem

Timeline:

```text id="z7j1qm"

User A

Reads balance = 100

User B

Reads balance = 100

User A

Writes 500

User B

Writes 700




Final result:



```plaintext
700

User A's update disappeared.

This is a concurrency problem.

Introducing Transactions

A transaction groups operations into one logical unit.

Example:

BEGIN;

UPDATE accounts

SET balance = balance - 100;

COMMIT;

A transaction provides:

Atomicity
Consistency
Isolation
Durability

The famous ACID properties.

Transaction Lifecycle

```text id="k2x8pq"

BEGIN

Perform Operations

COMMIT




Or:



```text id="m4v2zd"

BEGIN

 |

 |

Error

 |

 |

ROLLBACK

Transaction Manager

We add:

```rust id="p1q6lm"
pub struct Transaction {

pub id: u64,

pub state: State,

}




The database tracks active operations.

---

# Write-Ahead Logging (WAL)

What happens if the computer crashes?

Imagine:



```plaintext
Update balance

↓

Write to memory

↓

Power failure

The change disappears.

Databases solve this using WAL.

The WAL Principle

Before changing the database:

Write the intention to a log.

```text id="y9p4nv"

Transaction:

Change user 5 balance

Before:

1000

After:

900




Then:



```plaintext
Write log

↓

Modify database

WAL Architecture

```text id="a7x2qm"

Application

 |

 ▼

Transaction

 |

 ▼

Write Ahead Log

 |

 ▼

Database Pages




The log is the source of truth during recovery.

---

# WAL Implementation

A simple log entry:



```rust id="s8x9jk"
pub struct LogEntry {

    pub transaction_id: u64,

    pub page_id: u32,

    pub before:

    Vec<u8>,

    pub after:

    Vec<u8>,

}

Crash Recovery

Imagine:

```text id="c4q9hz"

Transaction starts

↓

Write WAL

↓

Update page

↓

Crash




On restart:

Database reads WAL:



```text id="f9v6kx"

Find unfinished transaction

↓

Redo changes

or

Undo changes

The database repairs itself.

Buffer Management Becomes More Important

Now we have:

Tables
Pages
Indexes
Transactions

Memory becomes critical.

Our buffer pool now manages:

```text id="r7n2wm"

          Buffer Pool

+--------------------------+

| Page 1 |

| Page 2 |

| Page 3 |

| Page 4 |

+--------------------------+




When memory is full:

Which page leaves?

---

# LRU Cache

Most databases use variations of:

Least Recently Used.

Idea:

Remove pages nobody is using.

Example:



```text id="v5b8df"

Recently Used

↓

Page A

Page B

Page C

↓

Old

Page D

Page D gets removed.

Updated TinyDB Architecture

Our database has grown.

```text id="r8x2kp"

                 SQL

                  |

                  ▼

            Query Engine

                  |

                  ▼

           Query Planner

                  |

      ┌───────────┴───────────┐

      ▼                       ▼

  Table Scan              Index Scan

      |                       |

      └───────────┬───────────┘

                  ▼

           Transaction Layer

                  |

                  ▼

             Buffer Pool

                  |

      ┌───────────┴───────────┐

      ▼                       ▼

    WAL                 Storage Engine

                              |

                              ▼

                            Disk




This is now starting to resemble a real database.

---

# What We Have Built

TinyDB now supports:

✅ Persistent storage

✅ Pages

✅ Tables

✅ SQL parsing

✅ Query execution

✅ B-Tree indexes

✅ Faster lookups

✅ Transactions

✅ WAL logging

✅ Crash recovery foundation

The system has evolved from a file writer into a real database engine.

---

# What Comes Next

There is still one major challenge.

Concurrency.

Right now, transactions exist, but they are still limited.

What happens when:

* Thousands of users read simultaneously?
* Multiple transactions update the same rows?
* Long-running queries overlap?
* Data changes while another transaction is reading?

Real databases solve this with **MVCC — Multi-Version Concurrency Control**.

In Part 4, we will build:

* Row versions
* Snapshots
* Isolation levels
* Garbage collection
* Concurrent reads and writes
* A PostgreSQL-style MVCC system

This is where TinyDB becomes truly powerful.

---

# Final Thoughts

Building indexes and transactions changed the way I think about databases.

A database is not simply a place where data lives.

It is an intelligent system constantly balancing competing goals:

* Speed
* Safety
* Consistency
* Reliability
* Scalability

An index sacrifices storage space to gain speed.

A transaction sacrifices simplicity to gain correctness.

A WAL sacrifices extra writes to gain durability.

Every database feature is a carefully engineered trade-off.

The beauty of database engineering is that there is rarely a perfect solution.

There are only intelligent compromises.

And understanding those compromises is what separates someone who uses databases from someone who can design them.





# Building Your Own SQL Database (Part 4)

## MVCC, Isolation, and Making TinyDB a Truly Concurrent Database

In the previous parts, we transformed TinyDB from a simple file storage system into a small relational database engine.

We started with bytes.

Then we built:

* Storage pages
* Tables
* Serialization
* Buffer pools
* SQL parsing
* Query execution
* Query planning
* B-Tree indexes
* Transactions
* Write-Ahead Logging

Our architecture now looks like this:



```text
                         SQL

                          │

                          ▼

                    Query Engine

                          │

                          ▼

                    Query Planner

                          │

          ┌───────────────┴────────────────┐

          ▼                                ▼

     Index Scan                      Table Scan

          │                                │

          └───────────────┬────────────────┘

                          ▼

                  Transaction Manager

                          │

                          ▼

                       Buffer Pool

                          │

              ┌───────────┴───────────┐

              ▼                       ▼

             WAL              Storage Engine

                                      │

                                      ▼

                                    Disk

But there is still a fundamental problem.

Our database works.

It stores data.

It executes queries.

It survives crashes.

But it struggles with something every modern database must handle:

Thousands of users doing things at the same time.

Imagine an online banking system.

One user checks their balance.

Another user transfers money.

A third user receives a payment.

All at the same time.

How does the database make sure everyone sees correct information?

This is where one of the most beautiful ideas in database engineering appears.

Multi-Version Concurrency Control.

MVCC.

The Concurrency Problem

Let's imagine a simple table.

Accounts

+----+---------+

| ID | Balance |

+----+---------+

| 1  | 1000    |

+----+---------+

Two transactions start.

Transaction A:

UPDATE accounts

SET balance = 900

WHERE id = 1;

Transaction B:

SELECT balance

FROM accounts

WHERE id = 1;

What should B see?

1000?

900?

Something else?

The database needs rules.

The Old Solution: Locks

The traditional approach is locking.

Transaction A locks the row.

Transaction A

       │

       ▼

   Lock Row

       │

       ▼

   Update Data

       │

       ▼

   Unlock

Transaction B waits.

Transaction B

       │

       ▼

    Waiting...

       │

       ▼

    Read Data

This works.

But it creates problems.

Why Locks Become Difficult

Imagine thousands of users.

User 1 ─┐

User 2 ─┤

User 3 ─┤── Waiting

User 4 ─┤

User 5 ─┘

The database becomes a traffic jam.

Readers block readers.

Readers block writers.

Writers block readers.

The system spends more time coordinating than working.

The MVCC Idea

MVCC takes a completely different approach.

Instead of changing data:

Create a new version.

Old data stays.

New data appears.

Example:

Before:

Account

Balance = 1000

After update:

Version 1

Balance = 1000


Version 2

Balance = 900

Both versions exist.

The database decides which one each transaction can see.

Data Becomes a Timeline

Traditional thinking:

Row = Current Value

MVCC thinking:

Row = History of Values

Example:

Time

────────────────────────────►


Version 1

Balance = 1000


        Update


Version 2

Balance = 900

The database becomes aware of time.

Adding Versions to Rows

Previously our row looked like:

pub struct Row {

    pub id: u64,

    pub data: Vec<u8>,

}

Now:

pub struct VersionedRow {

    pub id: u64,

    pub data: Vec<u8>,

    pub created_tx: u64,

    pub deleted_tx: Option<u64>,

}

Every row knows:

When it appeared
When it disappeared

Transaction IDs

Every transaction receives an identifier.

Example:

Transaction 1

Transaction 2

Transaction 3

Transaction 4

Rust:

pub struct Transaction {

    pub id: u64,

}

These IDs allow the database to understand history.

Creating a Snapshot

When a transaction begins:

BEGIN TRANSACTION

        │

        ▼

Create Snapshot

        │

        ▼

Execute Queries

The snapshot defines what data is visible.

Example Snapshot

Database:

Version 1

Name = Derek

Created = 1


Version 2

Name = Alex

Created = 5

Transaction 3 starts.

It sees:

Derek

Transaction 6 starts.

It sees:

Alex

Same database.

Different realities.

Visibility Rules

A version is visible if:

created_tx <= transaction_id

and:

deleted_tx > transaction_id

or:

deleted_tx does not exist

In simple terms:

"The row existed during this transaction."

Implementing Visibility

Rust:

impl VersionedRow {

    pub fn visible(

        &self,

        tx_id: u64

    ) -> bool {


        if self.created_tx > tx_id {

            return false;

        }


        match self.deleted_tx {

            Some(delete_id) => {

                tx_id < delete_id

            }


            None => true,

        }

    }

}

This tiny function is the heart of MVCC.

Updating Data in MVCC

Traditional database:

Modify existing row

MVCC:

Create new version

Example:

Before:

ID

1

Name

Derek

Update:

UPDATE users

SET name='Alex';

After:

Version 1

Derek

Created: 1

Deleted: 8



Version 2

Alex

Created: 8

Deleted: NULL

Nothing was overwritten.

Delete Operations

Deletion works the same way.

Traditional:

Remove row

MVCC:

Mark version invisible

Example:

Before:


User

Derek



After:


User

Derek

Deleted Transaction = 10

Old transactions can still see Derek.

New transactions cannot.

Reading With MVCC

A query:

SELECT *

FROM users;

becomes:

Read all versions

        │

        ▼

Check visibility

        │

        ▼

Return visible versions

Implementation:

rows
.iter()
.filter(
 |row|

 row.visible(transaction.id)

)

Handling Multiple Writers

Now imagine:

Transaction A:

Creates Version 5

Transaction B:

Creates Version 6

Both changes exist.

The database later decides whether conflicts exist.

Isolation Levels

Databases offer different levels of isolation.

The most common:

Read Uncommitted

You can see changes before they are committed.

Fast.

Dangerous.

Read Committed

Only committed data is visible.

Most common default.

Repeatable Read

A transaction sees the same data throughout its lifetime.

Serializable

Transactions behave as if executed one by one.

Safest.

Slowest.

Adding Isolation to TinyDB

Transaction state:

pub enum IsolationLevel {

    ReadCommitted,

    RepeatableRead,

    Serializable,

}

The visibility engine changes behavior depending on this choice.

MVCC Storage Layout

Our storage engine changes.

Before:

Page

 ├── Row

 ├── Row

 └── Row

After:

Page


 ├── Row Version

 │       └── Previous Version


 ├── Row Version

 │       └── Previous Version

History becomes part of storage.

Version Chains

Many databases organize versions as chains.

Example:

Newest

  │

  ▼

Version 3

  │

  ▼

Version 2

  │

  ▼

Version 1

The database walks backward until it finds a visible version.

Garbage Collection

A problem appears.

If every update creates a new version forever:

Derek

↓

Alex

↓

John

↓

Mary

↓

...

Storage grows endlessly.

We need cleanup.

Vacuuming

Databases periodically remove old versions.

Example:

Version 1

No transaction needs it

        │

        ▼

Delete

PostgreSQL calls this:

VACUUM

Implementing Simple Cleanup

We track active transactions.

pub struct TransactionManager {

    active:

    Vec<u64>,

}

Find the oldest active transaction:

Oldest Active = 50

Any version deleted before 50 can disappear.

The Complete Database Architecture

TinyDB is now a real database engine.

                         SQL

                          │

                          ▼

                    SQL Parser

                          │

                          ▼

                  Query Optimizer

                          │

                          ▼

                 Execution Engine

                          │

                          ▼

              Transaction Manager

                          │

              ┌───────────┴───────────┐

              ▼                       ▼

             MVCC                    WAL


              │                       │

              └───────────┬───────────┘

                          ▼

                    Buffer Pool

                          │

                          ▼

                  Storage Engine

                          │

                          ▼

                         Disk

What We Have Built

After four parts, TinyDB contains the fundamental ideas behind modern relational databases:

Storage

✓ Pages
✓ Serialization
✓ Tables
✓ Disk persistence

Query Processing

✓ SQL lexer
✓ Parser
✓ AST
✓ Query planner
✓ Execution engine

Performance

✓ Buffer pool
✓ B-Tree indexes
✓ Efficient lookups

Reliability

✓ Transactions
✓ WAL logging
✓ Crash recovery

Concurrency

✓ MVCC
✓ Snapshots
✓ Isolation levels
✓ Version cleanup

Comparing TinyDB With Real Databases

Feature	TinyDB	PostgreSQL
SQL Parser	Basic	Advanced
Storage Engine	Simple	Highly optimized
Indexes	B-Tree	Multiple types
Transactions	Yes	Yes
WAL	Basic	Advanced
MVCC	Basic	Advanced
Query Optimizer	Simple	Cost-based
Replication	No	Yes

The difference is not the fundamental ideas.

The difference is years of optimization.

Lessons From Building a Database

Building TinyDB taught me something important.

The hardest part of software engineering is rarely writing code.

The hardest part is managing complexity.

A database solves hundreds of difficult problems:

How do we store information?
How do we find it quickly?
How do we survive failures?
How do we support thousands of users?
How do we prevent corruption?

The answer is not one giant algorithm.

It is thousands of small engineering decisions working together.

Final Thoughts

A database is not just a storage system.

It is a carefully designed machine for managing reality.

It remembers the past through MVCC.

It protects the future through transactions.

It accelerates the present through indexes and caching.

Every SQL query is the result of decades of engineering research hidden behind a few simple words.

When you write:

SELECT * FROM users;

you are not just reading data.

You are interacting with:

A parser
A planner
An optimizer
A transaction system
A concurrency engine
A storage engine

Building your own database reveals the hidden world underneath everyday software.

And perhaps the biggest lesson is this:

Great systems are not built by making things complicated.

They are built by taking complicated problems and organizing them into simple, understandable layers.

Building Reliable UDP

Derek mwale — Mon, 03 Aug 2026 23:20:28 +0000

How to Turn an Unreliable Protocol into One You Can Trust

When most developers think about networking, they immediately think of TCP.

After all, TCP powers web browsing, APIs, databases, email, and countless distributed systems. It guarantees that data arrives in order, without duplication, and without corruption.

But there's another protocol that quietly powers some of the fastest systems in the world.

UDP.

Online multiplayer games.

Live video streaming.

Voice-over-IP.

DNS.

IoT devices.

Real-time telemetry.

These systems often choose UDP instead of TCP.

At first, that decision seems strange.

UDP doesn't guarantee delivery.

It doesn't guarantee order.

It doesn't prevent duplicates.

It doesn't even tell you whether the receiver exists.

It simply sends packets into the network and hopes for the best.

That sounds terrifying.

So why do engineers keep using it?

Because UDP gives us something incredibly valuable:

Control.

Instead of accepting TCP's built-in reliability mechanisms, we can design our own—optimized for our specific application.

When I first learned about reliable UDP, I assumed it was an entirely different protocol.

Then I discovered something fascinating.

Reliable UDP isn't a protocol.

It's an engineering pattern.

You're essentially rebuilding some of TCP's capabilities, but only the parts you actually need.

Let's build one from scratch.

Understanding UDP

Imagine writing a letter and throwing it into the wind.

Maybe it reaches its destination.

Maybe it doesn't.

That's UDP.

Application

      │

      ▼

UDP Socket

      │

      ▼

Network

      │

      ▼

Receiver

There is no handshake.

No confirmation.

No guarantee.

Why Would Anyone Use UDP?

Because guarantees come with costs.

TCP introduces:

Connection setup
Congestion control
Flow control
Retransmissions
Ordered delivery
Head-of-line blocking

Sometimes those features are exactly what we want.

Sometimes they're unnecessary.

Imagine an online shooter game.

If a player's position update is lost...

Sending the old position two seconds later is useless.

The next update will replace it anyway.

UDP excels in these situations.

The Problem

Suppose we send five packets.

Packet 1

Packet 2

Packet 3

Packet 4

Packet 5

The network delivers:

Packet 1

Packet 2

Packet 5

Packets 3 and 4 disappear.

UDP simply moves on.

Designing Reliable UDP

Instead of hoping every packet arrives, we'll build several mechanisms ourselves.

Our protocol will support:

Sequence numbers
Acknowledgements
Retransmissions
Timeouts
Sliding windows
Duplicate detection
Packet ordering

Together, these transform UDP into a dependable transport layer.

Packet Structure

We'll begin by designing our own packet format.

+--------------------------------+

| Sequence Number |

+--------------------------------+

| Acknowledgement |

+--------------------------------+

| Flags |

+--------------------------------+

| Payload |

+--------------------------------+

Every field serves a purpose.

Rust Representation

pub struct Packet {

    pub sequence: u32,

    pub acknowledgement: u32,

    pub flags: u8,

    pub payload: Vec<u8>,

}

Notice that we've added metadata on top of UDP.

This metadata is entirely ours.

Sequence Numbers

Every outgoing packet receives an increasing identifier.

Packet 1

Sequence = 1

------------------

Packet 2

Sequence = 2

------------------

Packet 3

Sequence = 3

Now the receiver knows exactly which packet it received.

Sending Packets

Our sender keeps track of the next sequence.

let packet = Packet {

    sequence: next_sequence,

    acknowledgement: 0,

    flags: 0,

    payload,

};

next_sequence += 1;

Simple.

But incredibly important.

Why Sequence Numbers Matter

Imagine receiving:

We immediately know:

Packet 7

Missing

Without sequence numbers, missing packets are invisible.

Acknowledgements

The receiver confirms successful delivery.

Sender

↓

Packet 10

↓

Receiver

↓

ACK 10

↓

Sender

Once acknowledged, the sender removes the packet from memory.

Packet Flow

Sender

Packet 12

──────────────►

Receiver

ACK 12

◄──────────────

Communication becomes two-way.

Detecting Lost Packets

Suppose this happens.

Packet 13

──────────────►

Lost

No acknowledgement arrives.

Eventually...

The sender concludes:

Packet Lost

Retransmission Timer

Every sent packet starts a timer.

Packet Sent

↓

Timer Starts

↓

ACK Received?

↓

Yes → Done

↓

No

↓

Resend Packet

This mechanism alone dramatically improves reliability.

Tracking Outstanding Packets

Rust.

use std::collections::HashMap;

let mut pending:

HashMap<u32, Packet>;

Whenever an ACK arrives:

pending.remove(&ack_number);

Only unacknowledged packets remain.

Timeouts

Choosing timeout values is surprisingly important.

Too short.

You resend unnecessarily.

Too long.

Recovery becomes slow.

Typical logic:

Send Packet

↓

Wait 500 ms

↓

ACK?

↓

No

↓

Retransmit

Real protocols adapt this dynamically.

Duplicate Packets

Networks sometimes duplicate packets.

Example.

Packet 20

↓

Receiver

↓

Packet 20

↓

Receiver

The receiver must ignore duplicates.

Duplicate Detection

Keep track of processed sequence numbers.

HashSet<u32>

When a packet arrives:

if received.contains(

    &packet.sequence

) {

    return;

}

Simple.

Effective.

Packet Ordering

Packets may arrive out of order.

Example.

Received

3

4

2

5

Applications usually expect:

We need a buffer.

Reordering Buffer

Incoming

↓

Buffer

↓

Sort

↓

Deliver

The application never sees disorder.

Sliding Window

Sending one packet at a time wastes bandwidth.

Instead, we'll allow multiple outstanding packets.

Window

+----------------------+

1

2

3

4

5

+----------------------+

As acknowledgements arrive, the window moves forward.

Sliding Window Example

Send

1

2

3

4

5

↓

ACK 1

↓

Window Advances

↓

Send 6

The network remains busy.

Throughput improves dramatically.

Sender State

pub struct Sender {

    next_sequence: u32,

    window_size: usize,

    pending:

    HashMap<u32, Packet>,

}

Everything revolves around outstanding packets.

Receiver State

pub struct Receiver {

    expected_sequence: u32,

}

Whenever packets arrive:

Expected

15

↓

Receive

15

↓

Deliver

↓

Expect

16

Simple state.

Huge impact.

A Complete Flow

Sender

Packet 1

────────────►

Receiver

ACK 1

◄────────────

Packet 2

────────────►

Lost

Packet 3

────────────►

Receiver

ACK 3

◄────────────

Timeout

↓

Resend Packet 2

────────────►

Receiver

ACK 2

◄────────────

Eventually every packet arrives.

Connection Setup

UDP has no connection.

We can create one.

Client

HELLO

────────────►

Server

WELCOME

◄────────────

Connected

This allows session management.

Heartbeats

How do we know the other side still exists?

Periodic heartbeat packets.

PING

────────────►

PONG

◄────────────

Miss several heartbeats.

Disconnect.

Congestion Awareness

Sending too fast overwhelms networks.

Our sender should monitor:

Packet loss
Round-trip time
Retransmissions

If loss increases:

Reduce sending rate.

Reliability isn't only about delivery.

It's also about respecting the network.

Putting Everything Together

                 Application

                       │

                       ▼

            Reliable UDP Layer

     ┌────────────┬─────────────┐

     ▼            ▼             ▼

Sequence      ACK Manager    Timers

     │            │             │

     ├────────────┼─────────────┤

     ▼            ▼             ▼

Retransmit   Sliding Window  Ordering

              │

              ▼

           UDP Socket

              │

              ▼

           Network

Notice how reliability emerges from multiple cooperating components.

Complete Architecture

             Application

                  │

                  ▼

       Reliable Transport API

                  │

                  ▼

      Packet Encoder / Decoder

                  │

      ┌───────────┼────────────┐

      ▼           ▼            ▼

 Sequence     ACK Logic    Retransmit

      │           │            │

      ├───────────┼────────────┤

      ▼           ▼            ▼

 Window      Ordering     Heartbeats

                  │

                  ▼

             UDP Socket

                  │

                  ▼

              Internet

Each layer has one responsibility.

That's good software architecture.

Where Real Protocols Go Further

Production systems extend these ideas significantly.

They include:

Adaptive congestion control
Forward Error Correction (FEC)
Packet compression
Encryption
Connection migration
Multipath routing
Flow control
Bandwidth estimation

Protocols like QUIC, which powers HTTP/3, are built on UDP and implement many of these ideas to provide secure, reliable, high-performance communication.

Lessons Learned

Building a reliable transport protocol taught me something broader about software engineering.

Reliability is rarely a single feature.

It's usually the result of many small mechanisms working together.

A sequence number alone isn't enough.

Acknowledgements alone aren't enough.

Timeouts alone aren't enough.

Sliding windows alone aren't enough.

But together, they create a communication layer that applications can trust.

That's a recurring theme in engineering.

Databases achieve reliability through transactions, logging, replication, and recovery.

Distributed systems combine consensus, retries, and idempotency.

Operating systems rely on schedulers, virtual memory, and interrupts.

Complex behavior emerges from simple, well-designed components.

Reliable UDP is another example of that principle.

Final Thoughts

One of the biggest misconceptions about UDP is that it's "unreliable."

In reality, UDP is simply minimal.

It gives developers a blank canvas.

You decide what reliability means.

Do you need every packet?

Do you care about order?

Can you tolerate duplicates?

How quickly should lost packets be retransmitted?

By answering those questions yourself, you can build transport protocols tailored to your application's needs instead of accepting one-size-fits-all behavior.

That's why technologies like QUIC, multiplayer game networking, live streaming platforms, and many distributed systems continue to embrace UDP.

Not because it's unreliable.

But because it gives engineers the freedom to build exactly the reliability they need.

And perhaps that's the most valuable lesson of all.

Sometimes the best engineering doesn't come from adding more features.

It comes from starting with something beautifully simple and carefully building only what your system truly requires.

MVCC Explained by Building One Yourself

Derek mwale — Mon, 03 Aug 2026 23:11:39 +0000

Understanding Multi-Version Concurrency Control from First Principles

Every database promises something deceptively simple.

It tells multiple users they can read and write data at the same time.

Your banking application updates balances while another customer checks their account.

An online store updates inventory while thousands of customers browse products.

A social media platform lets millions of people publish posts while everyone else continues scrolling.

From the outside, everything appears to happen simultaneously.

But underneath, things are far more complicated.

Imagine two people trying to edit the same document at the exact same moment.

Or two customers purchasing the last product in stock.

Or two bank transfers modifying the same account balance.

Without careful coordination, databases would constantly overwrite each other's work.

This is the problem of concurrency.

For decades, databases solved this using locks.

One transaction would lock a row.

Everyone else had to wait.

Locks work.

But they don't scale well.

Modern databases like PostgreSQL, MySQL (InnoDB), CockroachDB, Oracle, and others rely heavily on a different idea.

One that initially feels almost magical.

Instead of making readers wait...

They create multiple versions of the same data.

This technique is called Multi-Version Concurrency Control, or MVCC.

When I first encountered MVCC, it seemed surprisingly abstract.

After building one from scratch, however, it became one of the most elegant pieces of engineering I'd ever studied.

Let's build one together.

The Problem with Traditional Locks

Imagine a simple Users table.

Users

+----+---------+
| ID | Name    |
+----+---------+
| 1  | Derek   |
+----+---------+

Now suppose Transaction A wants to update the row.

UPDATE users
SET name='Alex'
WHERE id=1;

Meanwhile...

Transaction B wants to read it.

Traditional locking produces:

Transaction A

LOCK ROW

↓

Update

↓

Commit

↓

Unlock

↓

Transaction B Reads

The reader waits.

Not ideal.

Why Waiting Hurts Performance

Imagine one thousand users reading product information.

Only one administrator updates the description.

Without MVCC:

999 Readers

↓

Waiting

↓

Waiting

↓

Waiting

Even though readers don't modify data.

The database becomes unnecessarily slow.

The Core Idea Behind MVCC

Instead of modifying rows directly...

Create a new version.

Version 1

Name = Derek

↓

Update

↓

Version 2

Name = Alex

Readers continue using Version 1.

Writers create Version 2.

Nobody blocks.

Thinking in Timelines

MVCC is easier to understand visually.

Time

──────────────────────────►

Version 1

"Derek"

───────────────┐

               │

               ▼

Version 2

"Alex"

──────────────────────────►

Both versions coexist temporarily.

Every Row Has History

Instead of storing:

User

ID

Name

We store:

User

ID

Name

Created Version

Deleted Version

Each row knows when it became visible.

And when it disappeared.

Designing Our Row

Rust implementation.

pub struct Row {

    pub id: u64,

    pub name: String,

    pub created_tx: u64,

    pub deleted_tx: Option<u64>,

}

Notice something important.

Rows are never immediately overwritten.

Transactions Need IDs

Every transaction receives a unique identifier.

Transaction 1

↓

Transaction 2

↓

Transaction 3

↓

Transaction 4

Rust.

pub struct Transaction {

    pub id: u64,

}

Simple.

Yet incredibly powerful.

Reading Data

Suppose the table contains:

ID

Name

Created

Deleted

---------------------------------

1

Derek

1

None

Transaction 5 reads.

Is it visible?

Yes.

Because:

Created <= Transaction ID

The row already existed.

Updating Data

Now Transaction 6 executes:

UPDATE users

SET name='Alex'

WHERE id=1;

Instead of replacing the row...

We perform:

Old Row

Deleted = 6

↓

New Row

Created = 6

Now two versions exist.

Table After Update

+----+--------+---------+---------+

| ID | Name   | Created | Deleted |

+----+--------+---------+---------+

| 1  | Derek  |    1    |    6    |

| 1  | Alex   |    6    |   None  |

+----+--------+---------+---------+

Nothing was overwritten.

Visibility Rules

Now comes the magic.

Transaction 5 should see:

Derek

Transaction 7 should see:

Alex

Same table.

Different results.

Depending on when the transaction started.

Snapshot Isolation

Every transaction observes a snapshot.

Transaction Starts

↓

Snapshot Created

↓

Reads Always Use Snapshot

Even if newer updates occur.

This provides consistency.

Visualizing Snapshots

Time

────────────────────────────────────►

T1

Reads Derek

───────────────────────────

Update to Alex

───────────────────────────

T2

Reads Alex

Neither transaction blocks the other.

Implementing Visibility

Rust.

impl Row {

    pub fn visible(

        &self,

        tx: u64,

    ) -> bool {

        self.created_tx <= tx

        &&

        match self.deleted_tx {

            Some(d) => tx < d,

            None => true,

        }

    }

}

This small function powers MVCC.

Reading Rows

Query execution.

rows.iter()

.filter(

    |r|

    r.visible(tx.id)

)

Every query automatically filters invisible versions.

Applications never notice.

Inserting Rows

Insertion becomes simple.

Row {

    id,

    name,

    created_tx: tx.id,

    deleted_tx: None,

}

Only one version exists.

Initially.

Deleting Rows

Deletion doesn't remove anything immediately.

Instead:

Deleted = Transaction ID

The row remains.

Older transactions still need it.

Example Timeline

Transaction 2

Insert User

↓

Transaction 4

Delete User

↓

Transaction 3

Still Sees User

↓

Transaction 5

Doesn't See User

Notice the overlap.

Different transactions observe different realities.

Version Chains

Many databases organize versions as chains.

Newest

↓

Alex

↓

Derek

↓

Original

Readers traverse the chain until they find a visible version.

Storage Layout

Users

↓

Version List

↓

Row

↓

Previous Version

↓

Previous Version

Some databases optimize this differently.

The idea remains similar.

Garbage Collection

Eventually...

Old versions become unnecessary.

Version 1

No Active Transactions

↓

Delete Version

Otherwise storage would grow forever.

Vacuuming

PostgreSQL calls this process:

VACUUM

It removes obsolete versions once no active transaction can see them.

Transaction Manager

Eventually we need something managing transactions.

pub struct TransactionManager {

    next_id: u64,

}

Creating one.

pub fn begin(

    &mut self,

) -> Transaction {

    self.next_id += 1;

    Transaction {

        id: self.next_id,

    }

}

Every transaction receives a unique snapshot.

Building the Engine

Architecture.

Applications

        │

        ▼

Query Engine

        │

        ▼

Transaction Manager

        │

        ▼

MVCC Visibility

        │

        ▼

Storage Engine

Each layer performs one responsibility.

Reading Algorithm

Read Query

↓

Find Candidate Rows

↓

Check Visibility

↓

Return Matching Versions

Simple.

Elegant.

Fast.

Updating Algorithm

Find Row

↓

Mark Deleted

↓

Insert New Version

↓

Commit

No overwriting.

Ever.

Comparing Locking vs MVCC

Locking	MVCC
Readers Wait	Readers Continue
Writers Block Readers	Readers Never Block
Less Storage	More Storage
Simple	More Sophisticated
Lower Concurrency	Excellent Concurrency

The additional complexity buys significantly better scalability.

Real Database Behavior

PostgreSQL.

Tuple

↓

xmin

↓

xmax

These resemble our:

created_tx

deleted_tx

Different names.

Same philosophy.

Why This Changed My Thinking

Before understanding MVCC, I imagined databases as continuously modifying rows.

Now I think differently.

Rows don't simply exist.

They evolve through time.

A table becomes a timeline rather than a static structure.

Queries become historical observations.

Transactions become snapshots of reality.

That perspective fundamentally changed how I design backend systems.

Complete Architecture

                    Applications
                           │
                           ▼
                      SQL Queries
                           │
                           ▼
                    Query Planner
                           │
                           ▼
                  Transaction Manager
                           │
                           ▼
                    Snapshot Manager
                           │
                           ▼
                 Visibility Checker
                           │
          ┌────────────────┼────────────────┐
          ▼                ▼                ▼
     Version Chain    Version Chain    Version Chain
          │                │                │
          ▼                ▼                ▼
                  Storage Engine
                           │
                           ▼
                    Disk Persistence

Notice how MVCC isn't just one algorithm.

It's an architecture.

Implementation Summary

Our miniature MVCC engine now supports:

✓ Transaction IDs

✓ Snapshot visibility

✓ Multiple row versions

✓ Inserts

✓ Updates

✓ Deletes

✓ Version filtering

✓ Garbage collection strategy

Although simplified, these are the same foundational ideas used inside production databases serving millions of users every day.

Where Real Databases Go Further

Production systems extend these ideas considerably.

They include:

Write-Ahead Logging (WAL)
Crash recovery
Serializable isolation
Lock escalation
Index versioning
Replication
Distributed transactions
Checkpointing
Compression
Background vacuum workers

Yet none of those features replace MVCC.

They build upon it.

Final Thoughts

Building a miniature MVCC engine completely changed the way I think about databases.

Before studying concurrency control, I viewed a table as a collection of rows that were constantly being modified in place.

After implementing MVCC, I realized that's not how many modern databases think at all.

Instead of treating data as something that changes, they treat it as something that evolves.

Each update becomes a new version.

Each transaction becomes a snapshot in time.

Each query observes a carefully constructed view of reality that remains consistent from start to finish.

That's an incredibly elegant solution to one of the hardest problems in computer science.

It also reinforces a broader lesson about software engineering.

The fastest systems aren't always the ones that perform less work.

They're often the ones that organize work more intelligently.

Rather than forcing readers and writers to compete over the same data, MVCC lets them coexist.

Instead of serializing the world through locks, it embraces history.

And perhaps that's why MVCC remains one of my favorite database concepts.

It's a reminder that some of the best engineering solutions don't eliminate complexity.

They reorganize it into something that feels almost effortless.

The Things I Only Learned After Shipping 20+ Real Projects

Derek mwale — Sat, 01 Aug 2026 17:09:17 +0000

When I wrote my first real application, I thought the hardest part of software engineering was getting the code to compile.

Then I thought it was fixing bugs.

Later I thought it was learning new frameworks.

Today, after shipping more than twenty real projects—some successful, some abandoned, some open-sourced, some rewritten entirely—I think the hardest part of software engineering is something else.

It is learning how to make good engineering decisions.

Every project leaves behind code.

But more importantly, every project leaves behind lessons.

Some lessons arrive through success.

Others arrive through painful mistakes.

A few arrive months after deployment when you realize that the architecture you proudly designed now slows down every new feature.

Experience has a way of changing your priorities.

Earlier in my career, I optimized for writing more code.

Today, I optimize for writing better systems.

Looking back, there are many things I wish someone had told me before I started building production software.

Ironically, I probably wouldn't have understood them until I experienced them myself.

Software Is Never Finished

One of the biggest misconceptions I had was believing projects eventually become "done."

Reality looks very different.

Every application becomes a living system.

Users request new features.

Businesses change direction.

Dependencies receive updates.

Security vulnerabilities appear.

Performance expectations increase.

Operating systems evolve.

Browsers change.

Cloud providers introduce new services.

The software never stops moving.

The goal isn't to finish software.

The goal is to build software that can continue evolving.

That realization completely changed how I design systems.

Architecture Matters More Than Algorithms

University teaches algorithms.

Production teaches architecture.

I've rarely encountered applications that failed because they used the wrong sorting algorithm.

I've encountered many that struggled because responsibilities weren't clearly separated.

Business logic lived inside controllers.

Authentication spread across multiple services.

Database access appeared everywhere.

Dependencies became tangled.

Architecture quietly determines how easy software becomes to change.

Every Request Tells a Story

I no longer think about API endpoints individually.

I think about journeys.

```text id="r7n0av"
User Request
│
▼
Load Balancer
│
▼
Authentication
│
▼
API Controller
│
▼
Business Service
│ │
▼ ▼
Redis Cache PostgreSQL
│ │
└──────┬──────┘
▼
JSON Response




Every layer contributes.

When something becomes slow, the problem usually isn't one function.

It's the interaction between many components.

---

# Simplicity Ages Better

Earlier versions of my software often contained clever abstractions.

Generic builders.

Complicated inheritance.

Nested dependency chains.

They felt impressive.

Six months later they became difficult to understand.

Today I choose clarity instead.

Simple functions.

Predictable naming.

Small services.

Less abstraction.

Future maintainability matters more than present elegance.

---

# Databases Deserve More Respect

One lesson took years to appreciate.

The database isn't just where information lives.

It defines relationships.

Performance.

Consistency.

Integrity.

Good schemas simplify applications.

Poor schemas spread complexity throughout the entire codebase.

Indexes often improve performance more than rewriting application code.

Constraints eliminate entire categories of bugs.

The database quietly influences every request.

---

# Caching Is Architecture

Earlier I viewed Redis as an optimization.

Now I think of caching as part of system design.



```text id="x8h53m"
          Client Request
                 │
                 ▼
            Redis Cache
             │       │
           Hit      Miss
            │        │
            ▼        ▼
      Return Data Database
                     │
                     ▼
              Store Cache
                     │
                     ▼
               Return Data

Good caching isn't about speed alone.

It's about reducing unnecessary work.

The fastest query is often the one you never execute.

Events Changed How I Build Software

One architectural lesson transformed the way I think about large systems.

Instead of letting one service perform every responsibility, I let services communicate through events.

```text id="s4cvny"
User Registered
│
┌───────┼────────┐
▼ ▼ ▼
Email Analytics CRM Sync




The registration service no longer knows how emails work.

Or analytics.

Or customer relationship management.

It simply announces that something happened.

Other services react independently.

The architecture becomes remarkably flexible.

---

# Thin Controllers Saved Me Countless Hours

Earlier controllers handled everything.

Validation.

Business rules.

Emails.

Database logic.

Notifications.

Today my controllers barely do anything.



```rust id="fx1lh3"
pub async fn register(
    request: RegisterRequest,
    service: UserService,
) -> Result<ApiResponse<User>> {

    let user =
        service.register(request).await?;

    Ok(ApiResponse::success(user))
}

The service contains the actual business logic.

```rust id="q0m1ye"
impl UserService {

pub async fn register(
    &self,
    request: RegisterRequest,
) -> Result<User> {

    let user =
        self.repository
            .create(request)
            .await?;

    self.events.publish(
        UserRegistered {
            id: user.id
        }
    ).await?;

    Ok(user)
}

}




Small responsibilities produce maintainable systems.

---

# Monitoring Beats Guessing

Logging tells you what happened.

Monitoring tells you what's happening.

Earlier I waited for users to report bugs.

Today dashboards often identify problems before users notice them.

CPU usage.

Memory consumption.

Database latency.

Cache hit rate.

Request throughput.

Error percentages.

Observability transforms debugging into engineering.

---

# Testing Changes Design

One unexpected lesson surprised me.

Difficult-to-test code often indicates difficult-to-maintain architecture.

When services become modular...

Testing becomes easier.

When responsibilities become focused...

Unit tests become smaller.

Testing improves design almost as much as it verifies behavior.

---

# Failure Is Part of the Architecture

Production eventually teaches every engineer the same lesson.

Everything fails.

Networks timeout.

Servers restart.

Caches disappear.

Third-party APIs become unavailable.

Instead of asking:

"Will this fail?"

I now ask:

"How gracefully can it recover?"



```text id="c6fjvk"
 External API
      │
Available?
 │       │
Yes      No
 │        │
 ▼        ▼
Proceed  Retry
            │
            ▼
      Cached Result
            │
            ▼
       Log Failure

Reliable software anticipates failure instead of pretending it won't happen.

Documentation Saves Future You

One of my favorite engineers once said:

"Documentation is a gift to your future self."

I didn't appreciate that enough.

Months later, opening an old project without documentation feels like exploring an unfamiliar city.

Clear architecture diagrams.

API examples.

Database relationships.

Deployment instructions.

These reduce future confusion enormously.

Shipping Changes How You Learn

Tutorials teach techniques.

Production teaches judgment.

Books explain principles.

Real systems explain trade-offs.

Every decision becomes contextual.

Should we cache this?

Should we normalize this table?

Should this service own authentication?

Should this operation be asynchronous?

Experience teaches there is rarely one perfect answer.

Only thoughtful trade-offs.

Growth Changes Priorities

Earlier I cared about learning more programming languages.

Today I care more about designing better systems.

Languages matter.

Architecture matters more.

Frameworks evolve.

Engineering principles endure.

Clear responsibilities.

Loose coupling.

Simple interfaces.

Reliable monitoring.

Good naming.

Those ideas remain valuable regardless of technology.

Software Is Communication

Perhaps the biggest lesson surprised me.

Software isn't only communication between computers.

It's communication between engineers.

Naming communicates.

Architecture communicates.

Folder structures communicate.

API responses communicate.

Comments communicate.

The next engineer should understand your intentions without needing to ask.

Sometimes that next engineer is you.

Experience Made Me Slower—and Better

Interestingly, I write code more slowly today than I did years ago.

Not because I've forgotten anything.

Because I think longer before typing.

I sketch architectures.

Question assumptions.

Imagine future requirements.

Consider failure scenarios.

Good engineering begins long before implementation.

Final Thoughts

Looking back at more than twenty real projects, I don't remember every feature I built.

I don't remember every framework version.

I don't remember every library I used.

But I remember the lessons.

I remember the production bugs that revealed architectural weaknesses.

I remember the APIs that became difficult to evolve because I hadn't thought about versioning.

I remember the databases that taught me the value of proper indexing.

I remember the monitoring dashboards that identified issues before customers did.

I remember the projects that became easier to maintain because responsibilities were clearly separated.

Most importantly, I remember realizing that software engineering is far less about writing code than I once believed.

Code is only one expression of engineering.

Architecture.

Data modeling.

Caching.

Observability.

Documentation.

Testing.

Reliability.

Maintainability.

Those are equally important.

If I could go back and give my younger self one piece of advice before shipping the first project, it would be this:

Don't optimize for finishing software.

Optimize for living with it.

Because software stays with you far longer than the excitement of launching it.

Every design decision eventually returns.

Sometimes as technical debt.

Sometimes as simplicity.

Sometimes as confidence.

And after enough projects, you begin to realize that experience isn't measured by how many applications you've built.

It's measured by how many engineering lessons those applications quietly taught you along the way.

Caching Strategies Every Backend Engineer Should Know

Derek mwale — Sat, 01 Aug 2026 17:03:25 +0000

The first time I heard someone say, "The fastest database query is the one you never execute," I thought it was just another clever engineering quote.

Then I deployed applications into production.

I watched the same products being requested thousands of times.

I watched users refresh the same pages over and over.

I watched identical SQL queries hit the database every few milliseconds.

Suddenly the quote made perfect sense.

The problem wasn't that the database was slow.

The problem was that we kept asking it the same question.

That is where caching changes everything.

Many developers think of caching as an optimization that comes later. I used to think the same way.

Today I see it differently.

Caching is part of architecture.

It's one of the most effective ways to reduce latency, lower infrastructure costs, improve scalability, and create applications that continue performing well as traffic grows.

The interesting part is that caching isn't one technique.

It's an entire family of strategies, each solving a different problem.

Understanding when to use each one is one of the quiet skills that separates backend engineering from simply writing APIs.

Why Caching Exists

Every request consumes resources.

A database query uses CPU cycles.

Disk reads take time.

Network requests introduce latency.

External APIs may charge money.

If the same information is requested repeatedly, recomputing it every single time becomes wasteful.

Caching simply asks a better question:

Can we reuse work we've already done?

That single idea powers some of the fastest software systems in the world.

The Journey of a Request

Before discussing strategies, it's useful to understand where caches fit inside a backend architecture.

```text id="r0z5la"
Client
│
▼
Load Balancer
│
▼
API Gateway
│
▼
Redis Cache
│ │
Cache Hit Cache Miss
│ │
▼ ▼
Return Data Application
│
▼
PostgreSQL
│
▼
Store in Cache
│
▼
HTTP Response




Notice something important.

The database is no longer the first stop.

The cache becomes the first place the application looks.

That simple change can dramatically reduce database load.

---

# Strategy 1: Cache-Aside (Lazy Loading)

This is the strategy I use most often.

The application checks the cache first.

If the data exists, it returns immediately.

If not, it queries the database, stores the result in the cache, and returns it.



```text id="u2m8vh"
Incoming Request
       │
       ▼
Check Cache
   │        │
 Hit      Miss
  │         │
  ▼         ▼
Return   Database
             │
             ▼
      Save to Cache
             │
             ▼
        Return Data

Advantages

Simple to implement
Reduces database traffic
Only caches data that is actually requested
Works well for read-heavy applications

Disadvantages

First request is slower
Cache misses still reach the database

Rust Example

pub async fn find_product(
    &self,
    id: Uuid,
) -> Result<Product> {

    if let Some(product) =
        self.cache.get(id).await? {

        return Ok(product);
    }

    let product =
        self.repository.find(id).await?;

    self.cache
        .set(id, &product)
        .await?;

    Ok(product)
}

This implementation is remarkably small, yet it can eliminate thousands of unnecessary database queries every hour.

Strategy 2: Write-Through Cache

Instead of waiting for the next request, write-through updates the cache whenever data changes.

```text id="f5xqwp"
Application
│
▼
Update Database
│
▼
Update Cache
│
▼
Return Success




The cache and database remain synchronized.

Future reads are immediately available.

### Best for

* Frequently accessed data
* User profiles
* Product catalogs
* Configuration settings

---

# Strategy 3: Write-Behind (Write-Back)

Sometimes writing directly to the database becomes expensive.

Instead, the application writes to the cache first.

The cache later persists changes asynchronously.



```text id="k9yrab"
Application
      │
      ▼
 Write Cache
      │
      ▼
Return Success
      │
      ▼
Background Worker
      │
      ▼
Database

This approach dramatically improves write performance.

However, it requires careful handling to avoid losing data if the cache fails before persistence.

Strategy 4: Read-Through Cache

In this strategy, the application never talks directly to the database.

The cache becomes responsible for loading missing data.

```text id="t4mnqs"
Application
│
▼
Cache Layer
│ │
Hit Miss
│ │
▼ ▼
Data Load Database
│
▼
Store Cache
│
▼
Return Data




This keeps application code simpler because the cache handles loading automatically.

---

# Strategy 5: Distributed Caching

As applications grow, one cache server eventually becomes insufficient.

Distributed caches solve this problem.



```text id="g8lrcm"
          API Cluster
     ┌────────┼────────┐
     ▼        ▼        ▼
 App1     App2     App3
     │        │        │
     └────────┼────────┘
              ▼
      Redis Cluster
       │     │     │
       ▼     ▼     ▼
   Node1 Node2 Node3

Now cached data scales alongside the application.

This architecture powers many high-traffic platforms.

Strategy 6: HTTP Response Caching

Sometimes entire API responses can be cached.

```text id="v7yape"
Client
│
▼
CDN / Reverse Proxy
│ │
Hit Miss
│ │
▼ ▼
Response API Server




Content Delivery Networks (CDNs) often cache images, CSS, JavaScript, and even API responses.

The request never reaches your backend.

That means lower latency and reduced server costs.

---

# Cache Expiration Matters

A cache that never expires eventually becomes incorrect.

Every cached value should have a lifetime.



```text id="m3utfh"
Product Cache

Laptop

TTL: 10 Minutes

↓

Expired?

↓

Reload Database

Choosing the correct Time-To-Live (TTL) is an engineering trade-off.

Too short:

Frequent database queries.

Too long:

Stale information.

The right value depends on how often the underlying data changes.

Cache Invalidation Is Hard

One famous quote in software engineering says:

"There are only two hard things in Computer Science: cache invalidation and naming things."

The challenge is knowing exactly when cached information should be removed.

Suppose a product price changes.

The database updates.

If the cache isn't updated too, users continue seeing the old price.

That's why invalidation strategies are just as important as caching strategies.

Combining Multiple Cache Layers

Large applications rarely depend on a single cache.

```text id="q2ehik"
Browser Cache
│
▼
CDN Cache
│
▼
Reverse Proxy
│
▼
Redis
│
▼
Database




Each layer removes work from the layer below it.

This dramatically improves scalability.

---

# Monitoring Cache Performance

Adding a cache isn't enough.

Engineers also monitor its effectiveness.

Important metrics include:

* Cache hit ratio
* Cache miss ratio
* Memory usage
* Eviction rate
* Expiration frequency
* Response latency

A cache with a poor hit rate may simply consume memory without improving performance.

Monitoring reveals whether the strategy is actually working.

---

# Practical Example: Redis Integration

A simplified service might look like this.



```rust
pub async fn get_user(
    &self,
    id: Uuid,
) -> Result<User> {

    if let Some(user) =
        self.redis.get(id).await? {

        return Ok(user);
    }

    let user =
        self.repository.find(id).await?;

    self.redis
        .set_with_ttl(
            id,
            &user,
            600
        )
        .await?;

    Ok(user)
}

This implementation checks Redis first.

If no value exists, the database is queried.

The result is then cached for ten minutes before being returned.

Small code.

Massive impact.

When Not to Cache

Caching is powerful, but not everything benefits from it.

Avoid caching data that:

Changes every second
Contains highly sensitive user information without proper isolation
Must always reflect the latest state
Costs more to invalidate than to compute

Good engineering is knowing when not to optimize.

Lessons Experience Taught Me

Earlier in my career, I saw caching as something I could add after performance problems appeared.

Now I think differently.

The architecture itself should identify opportunities to reuse work.

If an endpoint is expected to receive thousands of identical requests, caching shouldn't be an afterthought.

It should be part of the design.

Experience also taught me that the best cache isn't always the largest one.

It's the one with the highest hit rate, the simplest invalidation strategy, and the clearest purpose.

Final Thoughts

Caching is one of those engineering concepts that appears deceptively simple.

Store some data.

Retrieve it later.

Avoid unnecessary work.

Yet beneath that simplicity lies one of the most influential techniques in backend engineering.

The world's busiest applications rely on carefully designed caching strategies to remain responsive under enormous traffic.

But the true value of caching extends beyond speed.

It protects databases from unnecessary load.

It reduces infrastructure costs.

It improves user experience.

It enables applications to scale gracefully.

Most importantly, it changes how we think about software.

Instead of asking, "How can I make this query faster?", we begin asking, "Do I need to execute this query at all?"

That shift in thinking is powerful.

It transforms optimization from making computers work harder to helping them work smarter.

And in many ways, that's what great backend engineering is all about—not doing more work, but designing systems that accomplish more by doing less.

How to Build APIs That Handle Millions of Requests

Derek mwale — Sat, 01 Aug 2026 16:59:10 +0000

When most developers build their first API, they focus on one question:

"Does it work?"

And that's exactly the right question to ask in the beginning.

Can users register?

Can they log in?

Can they retrieve data?

Can they update a resource?

At that stage, correctness matters more than performance.

But after shipping software into production, another question slowly becomes more important.

"Will it still work when millions of requests arrive?"

That is where software engineering becomes systems engineering.

Handling millions of requests isn't about writing one brilliant algorithm.

It isn't about choosing the fastest programming language.

It isn't about buying a larger server.

It's about designing an ecosystem where every component shares responsibility.

I've come to believe that scalable APIs aren't built through optimization alone.

They're built through architecture.

Scaling Begins Long Before Traffic Arrives

One of the biggest misconceptions in backend engineering is that scalability is something you add later.

It isn't.

Scalability is a design decision.

If an API depends on one overloaded database, no amount of optimization will save it.

If business logic is tightly coupled to HTTP requests, scaling becomes painful.

If every request performs unnecessary work, traffic eventually overwhelms the system.

Architecture determines scalability long before the first million requests ever arrive.

Understanding the Journey of a Request

Every request travels through many layers before reaching the user.

                    Client
                       │
                       ▼
                DNS Resolution
                       │
                       ▼
                 Load Balancer
                       │
                       ▼
                  API Gateway
                       │
                       ▼
            Authentication Layer
                       │
                       ▼
             Rate Limiting Check
                       │
                       ▼
             Application Service
                │            │
                ▼            ▼
           Redis Cache   PostgreSQL
                │            │
                └─────┬──────┘
                      ▼
              JSON Response
                      │
                      ▼
                    Client

When millions of requests arrive, every layer becomes important.

Optimizing only the controller misses the bigger picture.

Design Stateless APIs

One lesson that fundamentally changed how I design backend systems was understanding statelessness.

Each request should contain everything the server needs.

The server should not rely on memory from previous requests.

Instead of storing session data in application memory:

Server Memory

User A Session

User B Session

User C Session

I prefer:

Client
   │
   ▼
JWT Token
   │
   ▼
Any API Server

Now any server can process the request.

Adding more servers becomes trivial.

Horizontal scaling becomes possible.

Load Balancers Are the Unsung Heroes

One server can only do so much.

Eventually requests must be distributed.

                 Internet
                     │
                     ▼
              Load Balancer
         ┌────────┼─────────┐
         ▼        ▼         ▼
     API 1     API 2     API 3
         │        │         │
         └────────┼─────────┘
                  ▼
             Shared Database

The load balancer quietly becomes one of the most valuable components in the architecture.

Users rarely notice it.

But without it, growth becomes impossible.

Cache Before You Query

One lesson production taught me repeatedly:

Databases are expensive.

Not because they're slow.

Because unnecessary queries accumulate.

A cache avoids repeating work.

          Incoming Request
                 │
                 ▼
            Redis Cache
             │        │
           Hit       Miss
            │         │
            ▼         ▼
      Return Data   Database
                        │
                        ▼
                 Store in Cache
                        │
                        ▼
                 Return Response

If thousands of users request the same information every minute, the database shouldn't answer the same question thousands of times.

Caching transforms scalability.

Move Slow Work Into Background Jobs

Not everything belongs inside an HTTP request.

Suppose a user uploads an image.

The API could:

Resize it.

Generate thumbnails.

Send notifications.

Analyze metadata.

Store backups.

Or…

It could simply save the upload and return immediately.

        Upload Request
              │
              ▼
        Save Metadata
              │
              ▼
       Return Success
              │
              ▼
        Publish Event
              │
      ┌───────┼────────┐
      ▼       ▼        ▼
 Resize   Thumbnail   Notify
 Image     Service    Service

Users experience faster responses.

Servers process expensive work independently.

Everyone wins.

Separate Responsibilities

Large APIs often become difficult to scale because every service tries to do everything.

Today I prefer smaller responsibilities.

                  API Gateway
                       │
      ┌────────────────┼────────────────┐
      ▼                ▼                ▼
 User Service    Payment Service   Product Service
      │                │                │
      ▼                ▼                ▼
 Database        Payment API       Product DB

Services become easier to deploy.

Easier to monitor.

Easier to replace.

Growth becomes incremental instead of disruptive.

A Practical Rust Implementation

The controller should remain small.

pub async fn get_product(
    id: Uuid,
    service: ProductService,
) -> Result<ApiResponse<Product>> {

    let product =
        service.find(id).await?;

    Ok(ApiResponse::success(product))
}

The service handles caching before querying the database.

pub async fn find(
    &self,
    id: Uuid,
) -> Result<Product> {

    if let Some(product) =
        self.cache.get(id).await? {

        return Ok(product);
    }

    let product =
        self.repository.find(id).await?;

    self.cache
        .set(id, &product)
        .await?;

    Ok(product)
}

Notice what happens.

The database isn't contacted unless necessary.

Multiply this by millions of requests and the savings become enormous.

Protect the API With Rate Limiting

Scalability isn't only about serving more users.

It's also about protecting your infrastructure.

User Request
      │
      ▼
Rate Limiter
      │
  Requests > Limit?
      │        │
     No       Yes
      │        │
      ▼        ▼
 Continue   HTTP 429

Without limits, one client can unintentionally—or intentionally—consume disproportionate resources.

Healthy systems protect themselves.

Design Database Queries Carefully

The fastest API in the world cannot compensate for inefficient SQL.

Poor query:

SELECT * FROM orders;

Better query:

SELECT id, total
FROM orders
WHERE user_id = ?
LIMIT 20;

Even better:

Add indexes.

Avoid unnecessary joins.

Paginate results.

Return only required fields.

Database optimization often produces larger performance improvements than application optimization.

Pagination Is Mandatory

Collections grow.

Endpoints should expect that.

Instead of:

GET /users

Prefer:

GET /users?page=2&limit=50

Or cursor-based pagination.

Returning one million records in one response helps no one.

Monitor Everything

You cannot optimize what you cannot see.

Modern APIs should monitor:

Response time
Request count
Error rate
CPU usage
Memory usage
Cache hit ratio
Database latency
Queue length

Monitoring transforms intuition into evidence.

Think About Failure Before Success

Large systems eventually encounter failures.

Databases restart.

Redis disconnects.

External APIs timeout.

Design for resilience.

External Service
        │
Available?
 │         │
Yes       No
 │         │
 ▼         ▼
Continue  Retry
             │
             ▼
      Cached Response
             │
             ▼
      Error Monitoring

Graceful degradation keeps systems alive.

Horizontal Scaling Changes Everything

Eventually one server becomes many.

               Load Balancer
          ┌────────┼────────┐
          ▼        ▼        ▼
       Server1  Server2  Server3
          │        │        │
          └────────┼────────┘
                   ▼
          Shared Redis Cluster
                   │
                   ▼
           PostgreSQL Cluster

The application should not care which server processes the request.

Stateless design makes this possible.

Experience Changed My Perspective

Earlier in my career, I admired APIs with many features.

Today I admire APIs that remain calm under pressure.

Fast responses.

Predictable behavior.

Simple architecture.

Clear boundaries.

Reliable monitoring.

Graceful failure.

Scalability is rarely dramatic.

Most of the time it's the result of hundreds of small engineering decisions working together.

Final Thoughts

When people hear that an API handles millions of requests, they often imagine extraordinary hardware or revolutionary algorithms.

In reality, the secret is usually much quieter.

Requests are distributed instead of concentrated.

Frequently accessed data is cached instead of queried repeatedly.

Background jobs perform expensive work asynchronously.

Services have clear responsibilities.

Databases are carefully indexed.

Rate limiting protects infrastructure.

Monitoring reveals problems before users do.

Failures are anticipated instead of ignored.

The most scalable APIs I've encountered don't feel complicated.

They feel organized.

Every component understands its role.

Every request follows a predictable path.

Every optimization removes unnecessary work rather than adding unnecessary complexity.

Perhaps that's the most important lesson I've learned about scalability.

Building an API that survives millions of requests isn't about creating one incredibly powerful server.

It's about creating an architecture where thousands of ordinary requests can flow through ordinary components without overwhelming any single part of the system.

In the end, scalability is less about speed and more about balance.

When each layer does exactly what it should—and nothing more—the system continues to grow gracefully, one request at a time.

And that's the kind of backend architecture I aspire to build.

Things I Only Learned After Shipping Real Software

Derek mwale — Fri, 31 Jul 2026 13:48:53 +0000

There is a difference between building software that works and building software that survives.

I didn't fully understand that until I started shipping real applications.

At university, software felt predictable.

Projects ended when the assignment was submitted.

Requirements stayed the same.

Users behaved exactly as expected.

Servers never crashed.

Nobody tried to abuse the system.

Nobody entered invalid data.

Nobody clicked the same button twenty times.

Reality is very different.

Real software lives in an unpredictable world.

Users surprise you.

Businesses change direction.

Traffic spikes unexpectedly.

Databases fail.

Networks become unreliable.

Browsers behave differently.

APIs evolve.

Requirements never stop moving.

The biggest lessons I have learned as a software engineer didn't come from books.

They came after deployment.

Production became the greatest teacher I have ever had.

Shipping Is the Beginning, Not the Finish

Earlier in my career, I thought deployment was the final milestone.

Push the code.

Celebrate.

Move on.

Experience taught me something completely different.

Deployment is where learning begins.

Before launch, you're mostly making assumptions.

After launch, reality starts giving feedback.

Every user interaction becomes data.

Every support ticket reveals friction.

Every performance graph tells a story.

Software becomes a conversation between engineers and the real world.

Architecture Becomes Visible Under Load

Small applications can survive questionable architecture.

Large applications cannot.

The first hundred users rarely expose weaknesses.

The first hundred thousand usually do.

Features that seemed independent suddenly compete for resources.

Database queries multiply.

Caches become essential.

Background jobs accumulate.

Memory usage grows.

Latency increases.

Good architecture isn't tested when everything is quiet.

It is tested when everything becomes busy.

Every Request Is a Journey

One realization changed how I think about backend systems.

Every request travels through an entire ecosystem.

                User Request
                     │
                     ▼
               Load Balancer
                     │
                     ▼
              Authentication
                     │
                     ▼
              API Controller
                     │
                     ▼
             Business Service
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
     Redis Cache          PostgreSQL
          │                     │
          └──────────┬──────────┘
                     ▼
              Business Result
                     │
                     ▼
               HTTP Response

Earlier, I only focused on the controller.

Today I think about the entire journey.

Every component contributes to the user experience.

Monitoring Is More Valuable Than Logging

My first applications had log files.

My current applications have observability.

There's a difference.

Logs tell you what happened.

Monitoring tells you what is happening.

Dashboards answer questions before users ask them.

Which endpoint is slow?

Which database query is expensive?

How much memory is the application using?

What percentage of requests are failing?

Without visibility, debugging becomes guessing.

The Database Is Never Just Storage

Earlier, I thought databases stored information.

Now I think databases shape applications.

Poor schemas create complicated code.

Good schemas simplify everything.

Indexes affect performance.

Constraints prevent bugs.

Relationships define business rules.

The database quietly influences every API endpoint.

The longer I build software, the more respect I have for data modeling.

Users Never Behave the Way You Expect

Developers imagine ideal users.

Production introduces real users.

Someone uploads a 10 GB file.

Someone submits an empty form.

Someone refreshes the payment page repeatedly.

Someone disconnects halfway through a transaction.

Someone pastes emoji into a phone number field.

Reality is wonderfully creative.

Software must be equally resilient.

Failure Should Be Expected

One lesson arrived the hard way.

Everything eventually fails.

Networks timeout.

Servers restart.

Disks fill.

Third-party APIs disappear.

Caches expire.

Databases temporarily disconnect.

Instead of asking:

"Will this fail?"

I now ask:

"How should the system respond when it fails?"

Designing for Failure

Modern systems are built around graceful degradation.

          External API
               │
        Available?
        │       │
      Yes       No
       │         │
       ▼         ▼
Return Live   Return Cached
   Data         Response
       │         │
       └────┬────┘
            ▼
      Notify Monitoring

Failure isn't the exception.

It is another architectural scenario.

Thin Controllers, Smart Services

One thing I would never go back to is placing business logic inside controllers.

Today my controllers simply coordinate requests.

The real intelligence lives inside services.

pub async fn create_user(
    req: UserRequest,
    service: UserService
) -> Result<User> {

    service.create(req).await
}

The service contains the business rules.

impl UserService {

    pub async fn create(
        &self,
        request: UserRequest
    ) -> Result<User> {

        let user =
            self.repository
                .save(request)
                .await?;

        self.events
            .publish(UserCreated {
                id: user.id
            })
            .await?;

        Ok(user)
    }

}

Controllers stay simple.

Services remain reusable.

Maintenance becomes easier.

Background Jobs Matter More Than I Expected

Not everything belongs inside an HTTP request.

Emails.

Image processing.

Notifications.

Analytics.

Video transcoding.

Search indexing.

These belong elsewhere.

      User Uploads Image
               │
               ▼
        Save Metadata
               │
               ▼
        Return Success
               │
               ▼
      Publish Event
               │
     ┌─────────┴─────────┐
     ▼                   ▼
 Resize Image      Generate Thumbnail
     ▼                   ▼
 Store Image       Update Database

Queues dramatically improve responsiveness.

Users shouldn't wait for work that can happen later.

Caching Changes Everything

Caching used to feel like optimization.

Now it feels like architecture.

        Incoming Request
               │
               ▼
         Cache Lookup
         │         │
      Hit          Miss
       │            │
       ▼            ▼
 Return Data   Query Database
                    │
                    ▼
             Store in Cache
                    │
                    ▼
              Return Result

A well-designed cache can reduce database load by orders of magnitude.

Performance often comes from avoiding work, not performing it faster.

APIs Are Long-Term Contracts

Earlier I treated APIs as implementation details.

Today I treat them like products.

Every endpoint becomes a promise.

Changing field names affects clients.

Removing properties breaks integrations.

Consistency becomes more valuable than cleverness.

Versioning matters.

Documentation matters.

Predictability matters.

Shipping Changes Your Relationship With Code

Writing software feels creative.

Maintaining software feels educational.

Old code becomes documentation.

Earlier decisions become visible.

Naming choices matter.

Folder structures matter.

Architecture matters.

The code eventually teaches its author.

Sometimes gently.

Sometimes brutally.

The Most Valuable Feature Is Simplicity

One lesson keeps repeating.

Simple systems survive longer.

Simple APIs.

Simple deployment pipelines.

Simple naming.

Simple responsibilities.

Complexity accumulates naturally.

Simplicity requires intention.

Software Is Built by Feedback

The best ideas rarely arrive fully formed.

Users improve products.

Bug reports improve architecture.

Performance metrics improve infrastructure.

Production traffic improves scalability.

Feedback shapes software more than planning ever can.

Experience Changed My Definition of Success

Years ago, success meant finishing features.

Today, success means reducing future complexity.

Can another developer understand this?

Can I debug this six months later?

Can the system recover automatically?

Can it scale gracefully?

Can it evolve without breaking everything?

Those questions matter more now than they once did.

Final Thoughts

Looking back, I realize that the most important software engineering lessons were never written inside tutorials.

They were written inside production systems.

Every deployment challenged assumptions.

Every unexpected bug revealed a blind spot.

Every performance issue exposed architectural decisions I hadn't thought deeply enough about.

Every user interaction reminded me that software exists for people, not for developers.

Shipping real software taught me that architecture matters more than clever code.

Observability matters more than guesswork.

Resilience matters more than perfection.

Maintainability matters more than speed of implementation.

Feedback matters more than assumptions.

Most importantly, it taught me that software engineering is less about predicting the future and more about designing systems that can adapt to it.

No application is ever truly finished.

It grows.

It changes.

It teaches.

And if we're paying attention, every project leaves us a little wiser than the one before.

That's why I no longer see deployment as the end of development.

I see it as the moment the real education begins.

What I'd Build Differently Today

Derek mwale — Fri, 31 Jul 2026 13:39:40 +0000

One of the most valuable things software engineering has taught me is this:

The code I was most proud of five years ago is probably the code I would rewrite today.

Not because it was bad.

Not because it failed.

But because experience changes how you see systems.

Earlier in my career, I believed software engineering was mostly about writing code.

Today, I think it's mostly about making decisions.

Every project leaves behind lessons.

Some come from successful launches.

Others come from production incidents at two in the morning.

Some come from scaling problems.

Others come from maintaining code you wrote years ago.

Every application becomes a teacher.

Looking back at the systems I've built, I don't regret them.

I'm grateful for them.

They taught me exactly what I'd build differently today.

I Would Design the Architecture Before the Features

One of my earliest mistakes was thinking about features first.

Users can register.

Users can log in.

Users can upload files.

Users can purchase products.

These were useful goals.

But they weren't architectural goals.

Today I start somewhere else.

How will information flow?

Where does business logic belong?

Which components should know about each other?

How will this system grow in three years?

Features change.

Architecture remains.

From Requests to Information Flow

Every application is really a collection of information moving through a system.

Once I understood that, software became much easier to reason about.

Instead of asking:

"How do I implement login?"

I now ask:

"Where does authentication fit inside the architecture?"

Instead of asking:

"How do I upload images?"

I ask:

"How should media move through the system?"

The questions became architectural instead of functional.

The Architecture I'd Build Today

Instead of tightly coupling everything together, I'd organize the application into clear layers.

                    Modern Application Architecture

                  Client (Web / Mobile)
                           │
                           ▼
                    REST / GraphQL API
                           │
                           ▼
                 Authentication Middleware
                           │
                           ▼
                  Application Services
               ┌───────────┴───────────┐
               ▼                       ▼
          Domain Logic           Background Jobs
               │                       │
               ▼                       ▼
         Repository Layer         Event Queue
               │                       │
               ▼                       ▼
         PostgreSQL / Redis      Email / Analytics
                           │
                           ▼
                  Monitoring & Logging

Every layer has one responsibility.

The client presents information.

The API receives requests.

Services contain business logic.

Repositories manage persistence.

Queues handle asynchronous work.

Monitoring observes everything.

The architecture tells the story before the code does.

I Would Separate Business Logic Earlier

Earlier versions of my projects often placed too much logic inside controllers.

It worked.

Until it didn't.

Controllers became larger.

Testing became harder.

Features became interconnected.

Today I'd keep controllers incredibly small.

Their job is simple.

Receive a request.

Validate it.

Call a service.

Return a response.

The real intelligence belongs elsewhere.

A Simpler Service Layer

Instead of mixing everything together, I'd write services that solve one business problem.

pub struct OrderService {
    repository: OrderRepository,
    events: EventBus,
}

impl OrderService {

    pub async fn create_order(
        &self,
        order: Order
    ) -> Result<Order> {

        let saved = self.repository
            .save(order)
            .await?;

        self.events.publish(
            OrderCreated {
                id: saved.id
            }
        ).await?;

        Ok(saved)
    }
}

Notice what doesn't happen here.

No emails.

No notifications.

No analytics.

No inventory updates.

Those belong to independent services reacting to events.

Small responsibilities create maintainable systems.

I'd Embrace Events From Day One

One lesson experience taught me is that applications should communicate through events whenever possible.

Instead of this:

Order Service
      │
      ▼
Send Email
      │
      ▼
Update Inventory
      │
      ▼
Generate Invoice
      │
      ▼
Notify Analytics

I'd build this instead:

            Order Created Event
                    │
      ┌─────────────┼─────────────┐
      ▼             ▼             ▼
 Email Service Inventory Service Analytics
      │             │             │
      ▼             ▼             ▼
 Independent    Independent   Independent

Nothing depends directly on anything else.

Adding a new feature becomes publishing another subscriber.

Architecture becomes extensible instead of fragile.

I'd Invest More in Observability

Earlier in my career I logged errors.

Today I monitor systems.

Those are different things.

Modern software should answer questions automatically.

How many requests failed?

Which endpoint is slow?

Why is memory increasing?

Which customer experienced an error?

Monitoring transforms debugging from guessing into understanding.

I'd build dashboards before launching products.

Caching Would Be Part of the Design

Caching used to be something I added later.

Now I consider it during architecture.

             User Request
                  │
                  ▼
              API Gateway
                  │
          Cache Available?
           │             │
         Yes             No
           │             │
           ▼             ▼
     Return Cached    Query Database
        Response           │
                           ▼
                   Store in Cache
                           │
                           ▼
                    Return Response

Fast systems rarely rely only on faster databases.

They avoid unnecessary work.

I Would Think More About Failure

One question changed how I design systems.

"What happens if this component fails?"

Earlier versions of my software assumed success.

Modern systems assume failure.

Databases disconnect.

Networks timeout.

Caches disappear.

Queues overflow.

Users refresh pages.

Failures become architectural considerations instead of unexpected events.

Testing Would Begin Earlier

I once viewed testing as something performed after development.

Now I see testing as part of design.

When software is difficult to test...

It's often difficult to understand.

Writing testable software usually produces modular software.

The tests become documentation.

Future developers—including my future self—benefit.

Databases Deserve More Respect

Earlier projects treated the database like storage.

Today I see it as architecture.

Tables define relationships.

Indexes define performance.

Constraints preserve integrity.

Transactions preserve consistency.

A well-designed schema quietly improves every layer above it.

Poor schemas spread complexity everywhere.

I'd Build APIs Like Products

An API isn't simply an interface.

It's a contract.

Clients depend on it.

Future developers depend on it.

Documentation depends on it.

Consistency matters.

Versioning matters.

Naming matters.

Clear APIs reduce complexity before code executes.

Security Would Be Built In

Earlier projects often added security after features worked.

Now security becomes part of architecture.

Authentication.

Authorization.

Rate limiting.

Input validation.

Encryption.

Audit logging.

Security should never feel bolted on.

It should feel inevitable.

I'd Optimize for Maintenance

Earlier I optimized for finishing.

Today I'd optimize for maintaining.

Future developers matter.

Future teammates matter.

Future Derek matters.

Readable code.

Simple naming.

Predictable architecture.

Good documentation.

Maintenance is where software spends most of its life.

Continuous Delivery Over Perfect Releases

Another lesson took years to appreciate.

Shipping once isn't success.

Shipping continuously is.

I'd automate deployments.

Automate testing.

Automate monitoring.

Automate rollbacks.

Automation reduces human error.

Consistency scales better than heroics.

Software Is Never Finished

One belief changed everything for me.

Applications are living systems.

Users change.

Businesses evolve.

Technology advances.

Requirements shift.

Software must evolve alongside them.

Architecture should welcome change instead of resisting it.

Experience Changed My Priorities

Years ago, I admired clever code.

Today I admire understandable systems.

Years ago, I chased elegant algorithms.

Today I chase clear architecture.

Years ago, I celebrated large features.

Today I celebrate fewer responsibilities.

Experience didn't make me write more code.

It made me write less.

But with greater intention.

Final Thoughts

If I could start every project over again, I wouldn't begin by choosing a framework.

I wouldn't begin by designing database tables.

I wouldn't even begin by writing code.

I'd begin by understanding the problem deeply.

Then I'd design how information should move through the system.

I'd define clear boundaries between responsibilities.

I'd think about failure before success.

I'd build observability alongside functionality.

I'd treat APIs as long-term contracts.

I'd embrace events over tight coupling.

I'd automate everything that could be automated.

I'd optimize not only for today's requirements, but for the developer who will maintain the code years from now.

Ironically, I don't think I'd build dramatically different software.

Users might never notice the difference.

The interfaces would look similar.

The features would behave the same.

But underneath, the architecture would tell a very different story.

A story of loose coupling instead of dependency.

Clarity instead of cleverness.

Feedback instead of guesswork.

Maintainability instead of shortcuts.

And perhaps that's the greatest lesson software engineering has taught me.

The goal isn't to build software that impresses people on the day it's launched.

The goal is to build software that still makes sense years later, when requirements have changed, teams have grown, and someone—quite possibly your future self—has to understand every decision you made.

If there's one thing I'd build differently today, it isn't just the code.

It's the thinking behind the code.

Because in the end, architecture is simply experience made visible.

Teaching Machines to Recognize Patterns

Derek mwale — Fri, 31 Jul 2026 13:15:16 +0000

Teaching Machines to Recognize Patterns

When I first started learning about artificial intelligence, I thought machines were being taught to think.

The more I studied machine learning, the more I realized that wasn't quite true.

Machines aren't taught to think the way humans do.

They're taught to recognize patterns.

That distinction changed everything for me.

Whether it's recognizing handwritten digits, identifying tumors in medical images, detecting fraudulent transactions, recommending movies, or translating languages, modern AI isn't memorizing answers.

It's discovering patterns hidden inside data.

As software engineers, this should sound familiar.

Much of software engineering is also about patterns.

Design patterns.

Architectural patterns.

Communication patterns.

Database relationships.

User behavior.

Artificial intelligence simply extends this philosophy into data.

The more I explored machine learning, the more I realized that teaching machines to recognize patterns is one of the greatest engineering challenges of our time.

It combines mathematics, statistics, software engineering, distributed systems, and human intuition into one extraordinary discipline.

Every Intelligent System Starts With Data

Imagine trying to teach a child what a cat looks like.

You wouldn't begin with equations.

You would show examples.

One cat.

Another cat.

A black cat.

A white cat.

A kitten.

A large cat.

Eventually the child notices similarities.

Whiskers.

Ears.

Eyes.

Tail.

The child forms a mental model.

Machine learning follows exactly the same principle.

The difference is that machines don't see pictures.

They see numbers.

Every image becomes a matrix.

Every sentence becomes tokens.

Every sound becomes waveforms.

Teaching begins with representation.

Patterns Exist Everywhere

One realization fascinated me.

Patterns aren't unique to artificial intelligence.

Nature depends on them.

Weather follows patterns.

Economies follow patterns.

Languages follow patterns.

Music follows patterns.

Human behavior follows patterns.

Software follows patterns.

Artificial intelligence simply provides another way to discover them.

The engineer's job is to build systems capable of recognizing those hidden structures.

The Journey From Raw Data to Intelligence

Before a neural network can learn anything, data must travel through an engineering pipeline.

                Raw Data
                    │
                    ▼
        Data Collection
                    │
                    ▼
       Data Validation
                    │
                    ▼
       Data Cleaning
                    │
                    ▼
      Feature Extraction
                    │
                    ▼
      Machine Learning Model
                    │
                    ▼
      Pattern Recognition
                    │
                    ▼
        Prediction
                    │
                    ▼
      Continuous Learning

Notice something important.

The neural network occupies only one stage.

Everything surrounding it is equally important.

Good engineering creates good intelligence.

Representation Is the First Lesson

Machines cannot learn reality directly.

Reality must first be translated.

Images become pixels.

Words become embeddings.

Transactions become vectors.

GPS coordinates become numerical features.

One of the biggest lessons I've learned from AI is this:

Representation determines learning.

A poor representation makes recognition difficult.

A good representation makes learning almost effortless.

This principle extends beyond machine learning.

Well-designed databases are good representations of businesses.

Good APIs are good representations of communication.

Good software begins with good models.

Features Tell the Story

Early machine learning relied heavily on handcrafted features.

Engineers manually extracted edges.

Corners.

Textures.

Shapes.

Modern deep learning automates much of this work.

Yet the concept remains the same.

Features are descriptions.

The better we describe information, the easier patterns become to discover.

Imagine recognizing faces.

A neural network gradually learns:

Edges.

Curves.

Eyes.

Noses.

Mouths.

Entire faces.

Understanding emerges layer by layer.

Neural Networks Learn Hierarchically

One of my favorite characteristics of deep learning is hierarchical learning.

Simple patterns combine into complex patterns.

         Image

           │
           ▼

    Edge Detection

           │
           ▼

     Shape Detection

           │
           ▼

   Object Components

           │
           ▼

      Full Object

           │
           ▼

     Final Prediction

This reminds me of software architecture.

Functions combine into modules.

Modules combine into services.

Services combine into systems.

Complexity emerges from simple building blocks.

Every Prediction Is a Guess

A neural network never knows the answer with certainty.

It estimates.

Every prediction represents probability.

For example:

Dog      94%

Cat       4%

Fox       1%

Wolf      1%

The model isn't declaring absolute truth.

It's expressing confidence.

Understanding uncertainty is one of the most overlooked aspects of artificial intelligence.

Good engineering embraces probabilities instead of pretending certainty exists.

Learning Through Feedback

No pattern is recognized perfectly on the first attempt.

Learning requires repetition.

Input Data
     │
     ▼
Prediction
     │
     ▼
Compare Answer
     │
     ▼
Calculate Error
     │
     ▼
Backpropagation
     │
     ▼
Update Weights
     │
     ▼
Repeat

Each cycle slightly improves recognition.

Millions of iterations eventually produce remarkable accuracy.

A Simple Neural Network

At its heart, pattern recognition remains surprisingly elegant.

import torch
import torch.nn as nn

class PatternRecognizer(nn.Module):

    def __init__(self):

        super().__init__()

        self.network = nn.Sequential(

            nn.Linear(784,256),

            nn.ReLU(),

            nn.Linear(256,128),

            nn.ReLU(),

            nn.Linear(128,10)

        )

    def forward(self,x):

        return self.network(x)

Training is equally straightforward.

optimizer = torch.optim.Adam(model.parameters())

loss_function = nn.CrossEntropyLoss()

for images, labels in loader:

    prediction = model(images)

    loss = loss_function(prediction, labels)

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

Behind these few lines lies billions of mathematical operations.

The abstraction is one of modern software engineering's greatest achievements.

Convolutional Neural Networks Changed Vision

Traditional neural networks struggled with images.

Every pixel was treated independently.

Convolutional Neural Networks (CNNs) introduced a revolutionary idea.

Local patterns matter.

Instead of analyzing entire images at once, CNNs examine small regions.

Image

████████

Sliding Filter

■■■
■■■
■■■

↓

Edge

↓

Texture

↓

Shape

↓

Object

This dramatically improved computer vision.

Today CNNs power:

Medical imaging
Autonomous vehicles
Satellite analysis
Manufacturing inspection
Face recognition

Sometimes architectural improvements matter more than larger models.

Pattern Recognition Beyond Images

Images receive much attention.

Patterns exist everywhere.

Fraud detection recognizes unusual transactions.

Recommendation systems recognize preferences.

Language models recognize grammar and meaning.

Speech recognition identifies sound structures.

Predictive maintenance recognizes equipment failures.

AI isn't tied to one domain.

It recognizes structure wherever structure exists.

Engineering Reliable Pattern Recognition

Production AI involves more than training.

Engineers must monitor:

Prediction accuracy
Latency
False positives
False negatives
Data drift
Model drift
User feedback

An intelligent system that stops learning eventually becomes outdated.

Monitoring becomes another form of education.

Why More Data Isn't Always Better

One misconception fascinates me.

People often assume larger datasets automatically create smarter models.

Quality matters more.

Balanced examples.

Correct labels.

Representative samples.

Consistent preprocessing.

Clean information.

The best models often emerge from better engineering rather than bigger datasets.

Distributed Learning

Modern AI trains across many GPUs.

         Training Cluster

     GPU 1      GPU 2

       │          │

       └────┬─────┘

            ▼

 Gradient Synchronization

            │

            ▼

 Updated Global Model

Pattern recognition at scale becomes distributed systems engineering.

Networking.

Memory optimization.

Synchronization.

Fault tolerance.

Artificial intelligence increasingly resembles backend engineering.

Intelligence Is Generalization

Perhaps the most important lesson in machine learning is this.

The goal isn't memorization.

The goal is generalization.

A good model recognizes patterns it has never seen before.

This distinguishes intelligence from storage.

Anyone can memorize answers.

Recognizing new situations requires understanding.

Lessons Beyond Artificial Intelligence

Studying pattern recognition changed how I think about software.

I became more interested in relationships.

Representations.

Feedback.

Architecture.

Information flow.

Machine learning reinforced something I already suspected.

Good engineering isn't about handling isolated data.

It's about discovering meaningful structure.

Final Thoughts

Teaching machines to recognize patterns is one of the most fascinating engineering achievements of the modern era.

At first glance, it appears to be a story about algorithms.

Look closer, and it becomes a story about architecture.

About carefully designing systems that can transform raw information into meaningful understanding.

Every successful AI system begins long before a neural network makes its first prediction.

It starts with collecting reliable data.

Cleaning it.

Representing it.

Designing architectures that can discover hidden relationships.

Creating feedback loops that encourage continuous learning.

Monitoring performance.

Refining models.

Improving infrastructure.

The intelligence we admire is rarely the product of one brilliant algorithm.

It is the result of thousands of thoughtful engineering decisions working together.

Perhaps that is why artificial intelligence continues to inspire me.

Not because machines can recognize cats, translate languages, or recommend movies.

But because behind every one of those capabilities lies a timeless engineering principle.

Understanding emerges when we organize information well enough for patterns to reveal themselves.

Whether we are designing databases, building distributed systems, writing backend services, or training neural networks, we are ultimately solving the same problem.

We are helping systems recognize the patterns that matter.

And in many ways, that is the true art of building intelligent software.