DEV Community: Daniel Kuboi

Variational Autoencoders: Theory, Architecture, and Applications

Daniel Kuboi — Mon, 29 Jun 2026 12:07:00 +0000

Table of Contents

What Is a Variational Autoencoder?
The Problem VAEs Solve
Mathematical Foundations
Architecture Deep Dive
The ELBO Loss Function
Implementation: VAE on MNIST with TensorFlow/Keras
Visualising the Latent Space
Conditional VAE (CVAE)
Common Failure Modes and Fixes
VAEs vs GANs vs Diffusion Models
Real-World Use Cases
Further Reading

1. What Is a Variational Autoencoder?
A Variational Autoencoder (VAE) is a generative model that learns a compressed, continuous, and structured probabilistic representation of data. Introduced by Kingma & Welling in 2013, it sits at the intersection of deep learning and Bayesian inference.

Unlike a standard autoencoder — which maps inputs to a fixed-point embedding — a VAE maps inputs to a probability distribution over a latent space. This single design choice unlocks the ability to:

Generate new, coherent samples by sampling from the latent prior
Interpolate smoothly between data points
Disentangle factors of variation in the data
Detect anomalies by measuring reconstruction probability

Think of a VAE as a lossy compression algorithm that has been forced to be well-organised: similar inputs cluster together in latent space, and the space between clusters is meaningful rather than empty.

2. The Problem VAEs Solve

Standard Autoencoders Are Broken Generative Models

A standard autoencoder consists of an encoder E and decoder D:

x  →  E(x) = z  →  D(z) = x̂

The encoder collapses each input to a single point z. The problem is that the latent space is irregular — there is no guarantee that the region between two known encodings corresponds to anything meaningful. Interpolating between the encoding of a "3" and a "7" in MNIST may pass through a void that the decoder maps to noise.

VAEs force the encoder to predict a Gaussian distribution q(z|x) = N(μ, σ²) instead of a point. The decoder then reconstructs from a sample drawn from that distribution. Two regularisation pressures emerge naturally:

Reconstruction loss: The decoder must recover x faithfully.
KL divergence: The predicted distribution must stay close to a standard Normal N(0, I).

The tension between these two objectives shapes the latent space into something smooth, complete, and generative.

3. Mathematical Foundations

Generative Model

VAEs assume data x is generated by a two-step process:

z  ~  p(z)          = N(0, I)          (latent prior)
x  ~  p(x|z)        = Decoder(z)       (likelihood)

The true posterior p(z|x) is intractable (requires integrating over all z). VAEs approximate it with a learned distribution q_φ(z|x) — the encoder.

The KL Divergence Term

KL divergence measures how far q_φ(z|x) is from the prior p(z). For two Gaussians, it has a closed form:

KL(N(μ, σ²) || N(0, I)) = -½ Σ (1 + log σ² - μ² - σ²)

This is cheap to compute, differentiable, and pushes the encoder toward organised, zero-centred distributions.

The Reparameterisation Trick

Sampling z ~ N(μ, σ²) is not differentiable — gradients cannot flow through a stochastic node. The fix is elegant:

z = μ + σ ⊙ ε,    where ε ~ N(0, I)

Now ε is the randomness and μ, σ are deterministic outputs of the encoder. Gradients flow through μ and σ unobstructed.

4. Architecture Deep Dive

Encoder — maps x → (μ, log σ²). Two parallel output heads share a common backbone.

Sampling layer — implements the reparameterisation trick. No learnable parameters.

Decoder — maps z → x̂. Mirrors the encoder architecture. Output activation depends on data type (sigmoid for binary, linear for continuous).

β-VAE — a variant that scales the KL term by a factor β > 1, enforcing stronger disentanglement at the cost of reconstruction fidelity.

5. The ELBO Loss Function

The VAE is trained to maximise the Evidence Lower BOund (ELBO):

ELBO = E[log p(x|z)] - KL(q_φ(z|x) || p(z))

In practice this becomes a minimisation objective:

Loss = Reconstruction Loss  +  KL Loss
     = -E[log p(x|z)]       +  KL(q_φ(z|x) || p(z))

Reconstruction loss is typically:

Binary cross-entropy for image pixels in [0, 1]
Mean squared error for continuous data

KL loss uses the closed-form expression above.

The ELBO lower-bounds log p(x), so maximising it is equivalent to maximising the marginal likelihood of the data — the holy grail of unsupervised learning.

6. Implementation: VAE on MNIST with TensorFlow/Keras

6.1 Setup

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import matplotlib.pyplot as plt

# Reproducibility
tf.random.set_seed(42)
np.random.seed(42)

# Hyperparameters
LATENT_DIM   = 2      # 2D for easy visualisation; use 64–256 for real tasks
EPOCHS       = 30
BATCH_SIZE   = 128
BETA         = 1.0    # KL weight; increase for β-VAE disentanglement

6.2 Load and Preprocess Data

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Normalise to [0, 1] and flatten
x_train = x_train.astype("float32") / 255.0
x_test  = x_test.astype("float32")  / 255.0
x_train = x_train.reshape(-1, 784)
x_test  = x_test.reshape(-1, 784)

print(f"Train: {x_train.shape}  |  Test: {x_test.shape}")
# Train: (60000, 784)  |  Test: (10000, 784)

6.3 Sampling Layer (Reparameterisation Trick)

class Sampling(layers.Layer):
    """
    Reparameterisation trick:  z = μ + σ * ε,  ε ~ N(0, I)
    Allows gradients to flow through the stochastic sampling step.
    """
    def call(self, inputs):
        z_mean, z_log_var = inputs
        batch  = tf.shape(z_mean)[0]
        dim    = tf.shape(z_mean)[1]
        eps    = tf.random.normal(shape=(batch, dim))           # ε ~ N(0, I)
        return z_mean + tf.exp(0.5 * z_log_var) * eps          # z = μ + σ·ε

6.4 Encoder

def build_encoder(latent_dim: int) -> keras.Model:
    inputs   = keras.Input(shape=(784,), name="encoder_input")
    x        = layers.Dense(512, activation="relu")(inputs)
    x        = layers.Dense(256, activation="relu")(x)

    z_mean    = layers.Dense(latent_dim, name="z_mean")(x)
    z_log_var = layers.Dense(latent_dim, name="z_log_var")(x)
    z         = Sampling(name="z")([z_mean, z_log_var])

    return keras.Model(inputs, [z_mean, z_log_var, z], name="encoder")

encoder = build_encoder(LATENT_DIM)
encoder.summary()

6.5 Decoder

def build_decoder(latent_dim: int) -> keras.Model:
    inputs = keras.Input(shape=(latent_dim,), name="decoder_input")
    x      = layers.Dense(256, activation="relu")(inputs)
    x      = layers.Dense(512, activation="relu")(x)
    # Sigmoid output: pixels treated as Bernoulli probabilities
    outputs = layers.Dense(784, activation="sigmoid", name="decoder_output")(x)

    return keras.Model(inputs, outputs, name="decoder")

decoder = build_decoder(LATENT_DIM)
decoder.summary()

6.6 VAE Model with Custom Training Step

class VAE(keras.Model):
    def __init__(self, encoder, decoder, beta=1.0, **kwargs):
        super().__init__(**kwargs)
        self.encoder  = encoder
        self.decoder  = decoder
        self.beta     = beta

        # Metrics tracked across batches
        self.total_loss_tracker        = keras.metrics.Mean(name="loss")
        self.reconstruction_loss_tracker = keras.metrics.Mean(name="recon_loss")
        self.kl_loss_tracker           = keras.metrics.Mean(name="kl_loss")

    @property
    def metrics(self):
        return [
            self.total_loss_tracker,
            self.reconstruction_loss_tracker,
            self.kl_loss_tracker,
        ]

    def train_step(self, data):
        with tf.GradientTape() as tape:
            z_mean, z_log_var, z = self.encoder(data)
            reconstruction       = self.decoder(z)

            # Reconstruction loss: binary cross-entropy summed over pixels
            recon_loss = tf.reduce_mean(
                tf.reduce_sum(
                    keras.losses.binary_crossentropy(data, reconstruction),
                    axis=-1
                )
            )

            # KL divergence: closed-form for N(μ, σ²) vs N(0, I)
            kl_loss = -0.5 * tf.reduce_mean(
                tf.reduce_sum(
                    1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var),
                    axis=1
                )
            )

            total_loss = recon_loss + self.beta * kl_loss

        grads = tape.gradient(total_loss, self.trainable_variables)
        self.optimizer.apply_gradients(zip(grads, self.trainable_variables))

        self.total_loss_tracker.update_state(total_loss)
        self.reconstruction_loss_tracker.update_state(recon_loss)
        self.kl_loss_tracker.update_state(kl_loss)

        return {m.name: m.result() for m in self.metrics}

    def call(self, inputs):
        z_mean, z_log_var, z = self.encoder(inputs)
        return self.decoder(z)

6.7 Train

vae = VAE(encoder, decoder, beta=BETA)
vae.compile(optimizer=keras.optimizers.Adam(learning_rate=1e-3))

history = vae.fit(
    x_train,
    epochs=EPOCHS,
    batch_size=BATCH_SIZE,
    validation_data=(x_test, x_test),
    verbose=1
)

Expected output after 30 epochs

Epoch 30/30
469/469 ━━━━━━━━━━━━ 3s 6ms/step
loss: 162.4  recon_loss: 155.2  kl_loss: 7.2  val_loss: 165.1

Plot training curves

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

for ax, key in zip(axes, ["loss", "recon_loss", "kl_loss"]):
    ax.plot(history.history[key],     label="train")
    ax.plot(history.history[f"val_{key}"], label="val", linestyle="--")
    ax.set_title(key.replace("_", " ").title())
    ax.set_xlabel("Epoch")
    ax.legend()

plt.tight_layout()
plt.savefig("training_curves.png", dpi=150)
plt.show()

7. Visualising the Latent Space

7.1 Encode the Test Set and Plot μ

z_mean, _, _ = encoder.predict(x_test, batch_size=256)

plt.figure(figsize=(8, 6))
scatter = plt.scatter(z_mean[:, 0], z_mean[:, 1],
                      c=y_test, cmap="tab10", alpha=0.5, s=2)
plt.colorbar(scatter, label="Digit class")
plt.xlabel("z₁")
plt.ylabel("z₂")
plt.title("VAE Latent Space — MNIST Test Set")
plt.tight_layout()
plt.savefig("latent_space.png", dpi=150)
plt.show()

With a well-trained 2D VAE you will see the 10 digit classes arranged in a smooth, approximately circular manifold. Classes that look similar (3/8, 4/9) naturally cluster closer together.

7.2 Decode a 2D Grid of Latent Points

This reveals what the decoder has learned across the full latent space:

n = 20          # Grid size: 20×20 = 400 images
digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))

# Sample a grid of z values between the 5th and 95th percentiles
grid_x = np.linspace(-3, 3, n)
grid_y = np.linspace(-3, 3, n)[::-1]

for i, yi in enumerate(grid_y):
    for j, xi in enumerate(grid_x):
        z_sample    = np.array([[xi, yi]])
        x_decoded   = decoder.predict(z_sample, verbose=0)
        digit        = x_decoded[0].reshape(digit_size, digit_size)
        figure[
            i * digit_size : (i + 1) * digit_size,
            j * digit_size : (j + 1) * digit_size,
        ] = digit

plt.figure(figsize=(12, 12))
plt.imshow(figure, cmap="Greys_r")
plt.title("VAE Latent Space Manifold", fontsize=16)
plt.axis("off")
plt.tight_layout()
plt.savefig("latent_manifold.png", dpi=150)
plt.show()

The manifold plot is VAEs' signature strength: you can watch digits continuously morph into one another as you traverse the latent space. The transition from "1" to "7" passes through recognisable intermediate forms rather than noise.

7.3 Interpolation Between Two Images

def interpolate(vae, x_a, x_b, steps=10):
    """Linearly interpolate between two images via their latent means."""
    z_a, _, _ = vae.encoder(x_a[np.newaxis])
    z_b, _, _ = vae.encoder(x_b[np.newaxis])

    alphas = np.linspace(0, 1, steps)
    z_interp = np.array([(1 - a) * z_a + a * z_b for a in alphas])
    z_interp = z_interp.reshape(steps, LATENT_DIM)

    images = vae.decoder(z_interp).numpy()
    return images.reshape(steps, 28, 28)

# Interpolate between a "3" and an "8"
idx_a = np.where(y_test == 3)[0][0]
idx_b = np.where(y_test == 8)[0][0]

frames = interpolate(vae, x_test[idx_a], x_test[idx_b], steps=12)

fig, axes = plt.subplots(1, 12, figsize=(18, 2))
for ax, frame in zip(axes, frames):
    ax.imshow(frame, cmap="Greys_r")
    ax.axis("off")
plt.suptitle("Latent Interpolation: 3 → 8", fontsize=13)
plt.tight_layout()
plt.savefig("interpolation.png", dpi=150)
plt.show()

8. Conditional VAE (CVAE)

A standard VAE generates samples without control over their class. A Conditional VAE solves this by passing the label y to both encoder and decoder as a one-hot vector concatenated with the input.

NUM_CLASSES = 10

def build_cvae_encoder(latent_dim):
    x_in = keras.Input(shape=(784,))
    y_in = keras.Input(shape=(NUM_CLASSES,))

    h = layers.Concatenate()([x_in, y_in])
    h = layers.Dense(512, activation="relu")(h)
    h = layers.Dense(256, activation="relu")(h)

    z_mean    = layers.Dense(latent_dim)(h)
    z_log_var = layers.Dense(latent_dim)(h)
    z         = Sampling()([z_mean, z_log_var])

    return keras.Model([x_in, y_in], [z_mean, z_log_var, z], name="cvae_encoder")


def build_cvae_decoder(latent_dim):
    z_in = keras.Input(shape=(latent_dim,))
    y_in = keras.Input(shape=(NUM_CLASSES,))

    h = layers.Concatenate()([z_in, y_in])
    h = layers.Dense(256, activation="relu")(h)
    h = layers.Dense(512, activation="relu")(h)
    out = layers.Dense(784, activation="sigmoid")(h)

    return keras.Model([z_in, y_in], out, name="cvae_decoder")

With a trained CVAE, you can generate digit "7" on demand:

pythonlabel   = tf.one_hot([7], NUM_CLASSES)           # generate a "7"
z_sample = tf.random.normal(shape=(1, LATENT_DIM))
generated = cvae_decoder([z_sample, label])

9. Common Failure Modes and Fixes

Posterior Collapse

Symptom: KL loss drops to near zero early; the decoder ignores z entirely and generates blurry means.

Cause: The decoder is powerful enough to reconstruct x from context alone (common with autoregressive decoders).

Fixes:

KL annealing: start β = 0 and linearly ramp to 1 over the first few epochs
Use a Free Bits constraint: only penalise KL below a minimum threshold per dimension
Reduce decoder capacity

# KL annealing schedule
def kl_weight(epoch, total_epochs=30, warmup=10):
    if epoch < warmup:
        return epoch / warmup
    return 1.0

Blurry Reconstructions

Symptom: Reconstructions look washed out and lack fine detail.

Cause: Binary cross-entropy (or MSE) averages pixel uncertainty, producing the mean of all plausible images.

Fixes:

Use a more expressive decoder (convolutional, residual blocks)
Switch to a perceptual loss (VGG feature matching)
Use a flow-based or autoregressive decoder
Consider moving to a diffusion model for the highest quality

Mode Dropping
Symptom: The model generates only a subset of classes.

Fixes:

Check that batches are class-balanced
Increase β to prevent over-fitting to easy modes
Use a CVAE to give the model explicit class supervision

When to choose a VAE:
You need a compact, interpretable latent space
You need fast inference (real-time applications)
You need anomaly detection or data imputation
You need smooth interpolation between data points
You are working with structured/tabular data (not raw pixels)

11. Real-World Use Cases

Anomaly Detection

Compute the per-sample ELBO on held-out data. Samples with very low ELBO (high reconstruction loss + high KL) are anomalies.

def anomaly_score(vae, x, n_samples=50):
    """
    Monte Carlo estimate of -ELBO as anomaly score.
    Higher = more anomalous.
    """
    scores = []
    for _ in range(n_samples):
        z_mean, z_log_var, z = vae.encoder(x)
        x_recon = vae.decoder(z)

        recon = tf.reduce_sum(
            keras.losses.binary_crossentropy(x, x_recon), axis=-1
        )
        kl = -0.5 * tf.reduce_sum(
            1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var), axis=1
        )
        scores.append(recon + kl)

    return tf.reduce_mean(scores, axis=0).numpy()

Drug Discovery

VAEs are used to learn a continuous, differentiable latent space over molecular SMILES strings. Gradient-based optimisation in latent space — then decoding back to molecules — explores chemical property landscapes far more efficiently than random search.

*Recommendation Systems
*
Variational Autoencoders for Collaborative Filtering (VAE-CF) outperform matrix factorisation on sparse user-item interaction matrices by learning non-linear item representations and regularising via KL divergence.

Time-Series Imputation and Generation

VAEs naturally handle missing data: mask unknown timesteps in the reconstruction loss, and the model learns to impute from the latent distribution.

Text Generation (with modifications)

Text VAEs require careful treatment because discrete tokens break the reparameterisation trick. Solutions include the Gumbel-Softmax trick or a continuous approximation of the token embedding space.

Designing BuildBridge: A Large-Scale Construction Project Management Platform

Daniel Kuboi — Wed, 17 Jun 2026 09:38:49 +0000

1. What is BuildBridge?

BuildBridge is a construction technology platform that connects project owners, contractors, construction workers, material suppliers, transport providers, and administrators in a single ecosystem.

The platform enables:

Discovery of verified contractors
Digital contract signing
Real-time project communication
AI-powered material quotation
Construction materials marketplace
Transport and logistics coordination
Live project monitoring through photos and videos
Escrow-based milestone payments
Worker and supplier directories

Unlike traditional construction management that relies on phone calls, WhatsApp groups, spreadsheets, and paper contracts, BuildBridge provides transparency and accountability throughout the entire project lifecycle.

2. Requirements and Goals of the System

We will focus on the following requirements while designing BuildBridge.

Functional Requirements

User Management

Users should register and login.
Users should select roles.
Contractors should undergo verification.

Contractor Discovery

Search contractors using location.
Filter by experience, rating, trade and budget.
View contractor portfolios.
Send Requests for Quotations (RFQs).

Contract Management

Generate legal contracts.
Support digital signatures.
Store signed contracts.

Communication

Real-time chat.
File sharing.
Notifications.

AI Quotation

Generate material estimates.
Estimate project costs.
Convert quotation into purchase orders.

Marketplace

List materials.
Manage inventory.
Place orders.

Logistics

Book transportation.
Track deliveries.

Project Execution

Define project milestones.
Upload live photos and videos.
Approve or reject milestones.

Payments

Escrow deposits.
Milestone releases.
Payment tracking.

Non-Functional Requirements

Availability

99.9% uptime

Reliability

No contracts, payments, photos, or project records should be lost.

Performance

Contractor Search < 2 seconds

AI Quotation < 5 seconds

Chat Delivery < 500 milliseconds

Page Load < 3 seconds

Scalability

Support:

50,000 users in Year 1
500,000 users in Year 3
Millions of users long term

3. Design Considerations

BuildBridge differs from social networks because it is transaction-heavy rather than content-heavy.

The platform must support:

Heavy Writes

Photo uploads
Video uploads
Payment transactions
Chat messages
Contract updates

Heavy Reads

Contractor searches
Marketplace searches
Project dashboards
Material catalogs

High Reliability

Construction projects can involve millions of shillings.

Loss of:

Contracts
Payments
Milestone records

is unacceptable.

Trust

The platform's biggest challenge is establishing trust between project owners and contractors.

This is achieved through:

Escrow payments
Contractor verification
Live media uploads
Digital contracts

4. Capacity Estimation and Constraints

Let's estimate the scale of our system.

Users

Assume:

500,000 registered users

20% daily active users

500,000 × 20%
=
100,000 DAU

Projects

Assume:

50,000 active projects

Each project uploads:

20 photos daily

50,000 × 20
=
1,000,000 photos/day

Photo Storage

Average photo size:

5 MB

Storage per day:

1,000,000 × 5 MB
=
5,000,000 MB

=
5 TB/day

Storage per year:

5 TB × 365

=
1,825 TB

=
1.8 PB/year

Video Storage

Assume:

200,000 videos/day

Average video:

50 MB

Storage/day:

200,000 × 50 MB

=
10,000,000 MB

=
10 TB/day

Storage/year:

10 TB × 365

=
3.65 PB/year

Total Media Storage

Photos = 1.8 PB/year

Videos = 3.65 PB/year

Total

=
5.45 PB/year

Media storage becomes the largest cost in the system.

Chat Messages

Assume:

100,000 active users

Average:

50 messages/day

100,000 × 50

=
5,000,000 messages/day

Messages per second:

5,000,000

÷ 86,400

=
58 messages/sec

Peak traffic:

58 × 10

=
580 messages/sec

Payments

Assume:

50,000 transactions/day

Average project budget:

KES 2,000,000

Transaction volume:

50,000 × 2,000,000

=
KES 100 Billion/day

This requires enterprise-grade payment reliability.

5. High-Level System Design

At a high level we divide the system into independent services.

                    Users
                      |
                Load Balancer
                      |
                 API Gateway
                      |
---------------------------------------------------
|        |         |        |        |            |
Auth   Project   Search   Chat   Payment         AI
Service Service Service Service Service      Service
---------------------------------------------------
                      |
                  Kafka
               Event Bus
                      |
--------------------------------------------------------
|         |          |          |          |            |
Media   Contract  Marketplace Logistics Notifications Analytics
--------------------------------------------------------
                      |
------------------------------------------------
|              |              |                |
PostgreSQL   Redis      Elasticsearch       S3
------------------------------------------------

6. Database Schema

Users

Users
-----
UserID
Name
Email
Phone
Role
CreatedAt

Size estimation:

UserID         8 bytes
Name          50 bytes
Email         50 bytes
Phone         15 bytes
Role           4 bytes
Timestamp      8 bytes

Total

=
135 bytes

500,000 users:

500,000 × 135

=
67.5 MB

Projects

Projects
--------
ProjectID
OwnerID
ContractorID
Budget
Status
CreatedAt

Assume:

200,000 projects

200 bytes/project

200,000 × 200

=
40 MB

Media Metadata

Media

MediaID
ProjectID
URL
Latitude
Longitude
Timestamp
UploaderID

Approximate size:

300 bytes

Daily metadata:

1,200,000 uploads

× 300 bytes

=
360 MB/day

Yearly:

131 GB/year

Notice metadata is tiny compared to actual media storage.

7. Component Design

Authentication Service

Handles:

Registration
Login
JWT generation
OTP verification

User
 |
Auth API
 |
PostgreSQL
 |
JWT

Search Service

Handles:

Contractors
Workers
Shops
Materials

Instead of querying PostgreSQL directly:

PostgreSQL
     |
Kafka Events
     |
Elasticsearch
     |
Search API

This gives sub-second search.

Contract Service

Responsible for:

Contract generation
Version control
Digital signatures

Contract
    |
PDF Generator
    |
Digital Signature
    |
S3 Storage

AI Quotation Service

Input:

Building Type
Area
Floors
Quality Tier
Location

Output:

Material List
Quantities
Cost Estimate

Architecture:

User Input
      |
Validation
      |
Rules Engine
      |
Pricing Engine
      |
Quotation

8. Reliability and Redundancy

A project worth millions of shillings cannot lose data.

Every component should have replicas.

Primary Database
        |
Replication
        |
Secondary Database

Media Storage:

Photo
  |
S3 Bucket A
  |
Replicate
  |
S3 Bucket B

If one region fails, the other serves traffic.

9. Data Sharding

Suppose after expansion BuildBridge reaches:

20 million users

Single database becomes a bottleneck.

Sharding by ProjectID

Shard = ProjectID % 10

Example:

ProjectID = 145

145 % 10

=
5

Stored in:

Shard 5

Why Not UserID?

A large contractor may manage:

50,000 projects

This creates hotspots.

ProjectID distribution is more uniform.

10. Event-Driven Architecture

Whenever something happens:

Contract Signed

The system publishes:

ContractSigned Event

Kafka distributes this event to:

Notification Service
Analytics Service
Payment Service
Audit Service

Diagram:

Contract Service
        |
        v
      Kafka
        |
--------------------------
|      |       |         |
Notify Audit Analytics Payment

11. Escrow Payment Flow

This is the most critical business workflow.

Owner Deposits Money
          |
          v
     Escrow Wallet
          |
          v
Milestone Approved
          |
          v
Release Funds
          |
          v
Contractor

Example:

Project Budget:

KES 10,000,000

Foundation:

20%

Payment:

10,000,000 × 20%

=
KES 2,000,000

released after approval.

12. Caching and Load Balancing

Frequently accessed contractor profiles are cached.

User
 |
Redis Cache
 |
Hit?
 |
Yes -> Return
 |
No
 |
Database

Assume:

80% of searches target 20% of contractors.

Caching these records dramatically reduces database load.

13. Final Architecture


                 Users
                    |
             Load Balancer
                    |
              API Gateway
                    |
-----------------------------------------------------
|       |       |       |       |       |           |
Auth  Search Project Chat Payment AI Marketplace
-----------------------------------------------------
                    |
                 Kafka
                    |
-----------------------------------------------------
|        |         |        |        |              |
Media Contracts Logistics Notifications Analytics
-----------------------------------------------------
                    |
-----------------------------------------------------
|         |            |             |             |
Postgres Redis Elasticsearch     Object Storage
-----------------------------------------------------

Conclusion

BuildBridge is effectively a combination of several large-scale systems:

Uber for geolocation and discovery
Amazon for the marketplace
Stripe for escrow payments
DocuSign for digital contracts
Slack for communication
Jira for project execution tracking
AI-powered estimation systems for construction budgeting

The most important architectural decision is not the AI engine or marketplace. It is building a trustworthy ecosystem through contractor verification, digital contracts, milestone tracking, live geotagged evidence, and escrow payments. These features create the foundation upon which the entire BuildBridge platform is built.

The Power of Software Architecture

Daniel Kuboi — Thu, 11 Jun 2026 10:11:31 +0000

In more than half a century, a lot has changed in the world of technology—from vacuum tube computers to modern MacBooks with Core i7 processors, 16 GB of RAM, and 1 TB SSDs; from COBOL to modern object-oriented programming languages such as Java, Python, and C++. One component of technology that has experienced little change, however, is software architecture.

If Alan Turing were brought to the present day and asked to program in Python, he would probably spend a few weeks learning modern programming languages and tools before becoming comfortable building software applications. Likewise, a programmer from the 21st century would need only a few weeks to learn COBOL and the principles of programming on vacuum tube computers before becoming comfortable writing software. This demonstrates that the fundamental principles of software architecture have remained largely unchanged from 1952 to 2026.

Many people get confused by the terms Software Architecture and Software Design, often using the same description for both. They may be partially correct because the distinction between the two is subtle. Software architecture can be viewed as the high-level blueprint of a system, defining how modules, components, services, and layers are organized to build scalable and maintainable systems. Software design, on the other hand, focuses on the detailed implementation of those architectural decisions.

Many developers build software without considering software architecture, arguing that they are only creating a Minimum Viable Product (MVP) and that architectural improvements can come later. Unfortunately, this often does not happen. As competitors release new features, development teams become focused on keeping up with the market and have little time to address architectural problems.

As a result, organizations end up spending significant amounts of money maintaining and extending poorly structured systems. Each new release becomes increasingly expensive, often requiring additional software engineers to manage growing complexity. Eventually, the codebase becomes so difficult to maintain that releasing new features becomes a major challenge. This is often the beginning of a company's decline. Studies have shown that companies that incorporate software architecture into their development process spend less money on maintenance and new releases than companies that neglect it.

You may be wondering: How can I develop better software architecture? The answer is straightforward: write clean code, organize your software into well-defined modules, and design maintainable classes and functions. Below are some of the most important principles that should not be ignored when building software products intended to scale.

1. Separation of Concerns (SoC)

Divide the system into distinct features or layers, where each layer addresses a specific aspect of the problem. This ensures that business logic, database access, and user interface code remain separate and manageable.

2. Single Responsibility Principle (SRP)

Every component, module, or class should have one—and only one—reason to change. By limiting a unit's responsibilities, the codebase becomes significantly easier to read, test, and maintain.

3. Loose Coupling and High Cohesion

Components should depend on each other as little as possible (loose coupling). At the same time, the internal elements of a module should work closely together to achieve a single, well-defined purpose (high cohesion).

4. Abstraction and Encapsulation

Hide the complex internal workings of a component behind clear and stable interfaces. Consumers of the component should understand what it does without needing to know how it does it.

5. Don't Repeat Yourself (DRY)

Eliminate redundancy by ensuring that every piece of knowledge or logic has a single, unambiguous representation within the system. Shared logic should be generalized and reused rather than copied.

6. YAGNI (You Aren't Gonna Need It)

Avoid adding functionality, abstractions, or complex architectural patterns until they are genuinely necessary. Designing for hypothetical future requirements often introduces unnecessary complexity.

7. Dependency Inversion

High-level business modules should not depend directly on low-level implementation details. Instead, both should depend on abstractions, making the system more adaptable to changes in technologies, frameworks, or databases.

8. Keep It Simple, Stupid (KISS)

Prioritize simplicity in both design and implementation. Simple systems are easier to understand, maintain, and debug than unnecessarily complex ones.

Conclusion

Software architecture is a critical part of building scalable and maintainable software products. While investing in good architecture may seem expensive in the short term, the cost of poor architecture is almost always much higher.

If you think software architecture is expensive, try bad architecture.

Understanding System Design

Daniel Kuboi — Wed, 10 Jun 2026 14:36:35 +0000

In this article we will go through the process involved when designing scalable systems. Here are some questions for designing that should be answered before moving on to the next steps:

Will users be able to use post request faster
Will users of our service be able to post tweets and follow other people?
Should we also design to create and display the user’s timeline?
Will tweets contain photos and videos?
Are we focusing on the backend only or are we developing the front-end too?
Will users be able to search tweets?
Do we need to display hot trending topics?
Will there be any push notification for new (or important) tweets?

The next step will be to define what APIs are expected from the system. This will not only establish the exact contract expected from the system, but will also ensure if we haven’t gotten any requirements wrong.
It is always a good idea to estimate the scale of the system we’re going to design. This will also help later when we will be focusing on scaling, partitioning, load balancing and caching.

What scale is expected from the system
How much storage will we need?
What network bandwidth usage are we expecting? This will be crucial in deciding how we will manage traffic and balance load between servers. Next is defining data models early will clarify how data will flow among different components of the system. Later, it will guide towards data partitioning and management. how they will interact with each other, and different aspect of data management like storage, transportation, encryption, etc. Which database system should we use? Will NoSQL like Cassandra best fit our needs, or should we use a MySQL-like solution? What kind of block storage should we use to store photos and videos? Next step is implementing a high level design where you draw 5 – 6 core components of the applications

The next step will be defining detailed design on specific areas or components of the application, and also providing the PROS and CONS for why you prefer certain approach over the other
The tradeoffs maybe:

Should we partition our data to distribute it to multiple databases? Should we try to store all the data of a user on the same database? What issue could it cause?
Should we try to store our data in such a way that is optimized for scanning
How much and at which layer should we introduce cache to speed things up?
What components need better load balancing?

Identifying and resolving bottlenecks is also critical in designing scalable systems, this help in developing system that is fault tolerant and can survive under critical conditions

Is there any single point of failure in our system? What are we doing tomitigate it?
Do we have enough replicas of the data so that if we lose a few servers we can still serve our users?
Similarly, do we have enough copies of different services running such that a few failures will not cause total system shutdown?
How are we monitoring the performance of our service? Do we get alerts whenever critical components fail or their performance degrades?

Implementing Cookies with Nest.js

Daniel Kuboi — Wed, 03 Jun 2026 09:29:45 +0000

An HTTP cookie is a small piece of data stored by the user's browser. Cookies were designed to be a reliable mechanism for websites to remember stateful information. When the user visits the website again, the cookie is automatically sent with the request.

Before implementing cookies in nest.js, first is to install required packages and it's typescript definitions

$ npm i cookie-parser
$ npm i -D @types/cookie-parser

Once installation is complete, set cookie-parser middleware as global middleware in the main.ts file of the application.
In the src/main.ts file add cookie-parser

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import * as cookieParser from 'cookie-parser';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Enable cookie parsing globally
  app.use(cookieParser('your-secret-key-here')); 

  await app.listen(3000);
}
bootstrap();

You can pass several options to the cookieParser middleware:

secret a string or array used for signing cookies. This is optional and if not specified, will not parse signed cookies. If a string is provided, this is used as the secret. If an array is provided, an attempt will be made to unsign the cookie with each secret in order.
options an object that is passed to cookie.parse as the second option. See cookie for more information.

The middleware will parse the Cookie header on the request and expose the cookie data as the property req.cookies and, if a secret was provided, as the property req.signedCookies. These properties are name value pairs of the cookie name to cookie value.

When a secret is provided, this module will unsign and validate any signed cookie values and move those name value pairs from req.cookies into req.signedCookies. A signed cookie is a cookie that has a value prefixed with s:. Signed cookies that fail signature validation will have the value false instead of the tampered value.

To issue a cookie, inject the underlying platform Response object using the @Res() decorator.

import { Controller, Get, Res } from '@nestjs/common';
import { Response } from 'express';

@Controller('auth')
export class AuthController {
  @Get('login')
  setCookie(@Res({ passthrough: true }) response: Response) {
    response.cookie('accessToken', 'xyz123fakeToken', {
      httpOnly: true,     // Prevents client-side scripts from reading the cookie
      secure: true,       // Ensures cookie is sent only over HTTPS
      sameSite: 'strict', // Controls cross-site request behavior
      maxAge: 3600000,    // Expires in 1 hour (milliseconds)
    });

    return { message: 'Logged in and cookie issued successfully!' };
  }
}

To read incoming cookies, use the standard @Req() decorator to access the parsed properties. Alternatively, you can create a custom param decorator for cleaner code.

import { Controller, Get, Req } from '@nestjs/common';
import { Request } from 'express';

@Controller('profile')
export class ProfileController {
  @Get()
  getProfileCookies(@Req() request: Request) {
    // Read unsigned cookies
    const normalCookie = request.cookies['accessToken'];

    // Read signed cookies (if you provided a secret to cookieParser)
    const signedCookie = request.signedCookies['accessToken'];

    return { normalCookie, signedCookie };
  }
}

To delete a cookie from the user's browser, match the exact configuration options used to set it (like path or domain) and call .clearCookie():

@Get('logout')
logout(@Res({ passthrough: true }) response: Response) {
  response.clearCookie('accessToken', {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
  });
  return { message: 'Logged out successfully!' };
}

Now, your application can decode incoming cookies and add them to the corresponding req object at req.cookies.

Why Page Weight Matters: Aim For < 500kb

Daniel Kuboi — Tue, 02 Jun 2026 11:53:46 +0000

Why Page Weight Matters: Aim For < 500kb
Keeping page weight small is critical in making your application loading faster. Here are techniques used to keep page weight small:

Image Optimization
Images are generally the largest contributor to page bloat. There are a few things you should be doing to optimize your images.
Increasing compression levels, image file sizes can be dramatically reduced with little or no noticeable loss in visual quality. Background images are often the best candidates for optimization because they tend to be large and unnecessarily high quality.
Tools like Guetzli that generates JPEG images with file sizes 35% smaller than current alternatives.
Modern compression tools and formats continue to improve, allowing developers to achieve smaller file sizes, faster page loads, lower bandwidth usage, and a better user experience.
If using pallete or indexed color image, choosing the optimized pallette can also help reduce file size using image manipulation softwares.
Depending on your image, a PNG8 will often be fine instead of a PNG24, and will result in a significantly smaller file.
A further optimisation that can be performed is to strip EXIF image metadata. EXIF (Exchangeable Image File Format) metadata contains information such as date and time of image capture, camera settings, copyright information, and even geolocation information (e.g. GPS coordinates).
You can use simple free tools such as ImageOptim, PNG Gauntlet, JPEG Optimizer, and TinyPNG to quickly compress and optimize your images, as well as more comprehensive programs such as GIMP and Photoshop.
Using the right format for your images in your application is also important. WebP and SVG are the most recommended.
Images should be resized to the size that they will be displayed at. If you want to display an image at 200x200 pixels then don’t send a 400x400 pixel image and let the browser resize; send a 200x200 pixel image and save bandwidth.

Lazy loading
It mean image is not laoded until is needed
If an image is not in the viewport, then it can’t be seen by a user, and if the user never scrolls down, then the image will never be displayed, so loading it is a waste of bandwidth.
Making a good optimization of it will help reduce size of byte downloaded when only a portion of a page is loaded
Lazy-loading also helps to improve the perceived performance by making the page feel snappier since the initial payload is not as large.
Consider whether you really need a background image! CSS can do a really good job if you just need a fade or simple repeating pattern. While this won't be the right thing to do in every case—for example a small image vs a complicated CSS effect - it will save you some bytes in some cases.

GZIP Server Compression
All modern web servers can compress the text files that make up your website, such as HTML, JavaScript, CSS, before they are sent to the browser. This reduces the amount of data that needs to be sent to the browser, resulting in really big performance gains with no downside.

Magnification
Minification (or minimization) is the process of reducing the size of text-based files by removing unnecessary characters such as whitespace, newline characters, and comments. File size can drop by over 30%. JavaScript, CSS, HTML and SVG can all be minimized.
Build tools such as Gulp and Grunt can also automate these tasks as part of your build process.

Caching
Caching reduces the number of requests made to a server. It works by downloading pages and resources only on the first request, and won’t download anything a second time unless it has changed, or until a certain amount of time has passed. By setting cache headers, you can specify how long a specific resource should remain cached. The next time the browser needs it, it can look in the cache to see if it has it already. Because the browser has less to download, pages load more quickly.
HTML5 offers a number of APIs that give the developer far more control over cached resources. AppCache and Service Workers are two APIs that give enough control over cached resources and HTTP requests that reliable caching and offline experiences can be built.

Avoid unecessary embeds
Things like YouTube videos, embedded Twitter tweets, and social media icons can all harbour unexpected bloat. For example:
• YouTube video embed: adds an extra 574KB. You might think that some of this is the placeholder image it shows before you play the video but no, that image is generally only a fraction of the embed itself. Unfortunately this cost is incurred whether the user hits play or not.
• Twitter tweet embed: adds 74KB for a 140B tweet. That’s a pretty appalling ratio of content to packaging. 140bytes in a 75,766byte package, or 0.1% of transfer weight is the text payload.
• AddThis social media buttons: adds 248KB for simple social media sharing buttons. That's quite a payload for questionable value.

Speed is also a Feature

Daniel Kuboi — Mon, 01 Jun 2026 12:11:52 +0000

Modern web applications are more powerful than ever. Frameworks keep evolving, AI can generate code in seconds, and developers can ship features faster than at any other point in history. Yet despite all this progress, many applications still feel frustratingly slow. We've all experienced it. You click a link and stare at a blank screen. You press a button and wonder whether it actually worked. You try to scroll, but the page stutters and freezes. At that moment, users aren't thinking about your architecture, design patterns, or technology stack. They're thinking about one thing:

"Why is this so slow?"
That's why performance isn't just a technical concern. It's a product feature.

Users may never notice a well-designed database schema or a carefully structured codebase, but they immediately notice when an application feels fast. Speed creates confidence. It makes software feel reliable, polished, and trustworthy.

In many cases, speed is the very first feature users experience.

This article explores web performance from the user's perspective. We'll look at how people perceive speed, learn how modern performance metrics measure real-world experiences, understand the RAIL model, and discuss practical techniques for building applications that feel fast on every device and network.

Why Performance Matters
Performance is often discussed as a technical metric, but at its core, it's about people.

When users visit your application, they're trying to accomplish something. Maybe they're shopping online, reading an article, sending a message, or managing their finances. They don't care how many microservices are running behind the scenes. They care about completing their task quickly and without frustration.

Research has consistently shown that delays have a direct impact on user behavior. The longer people wait, the more likely they are to leave.

Human perception generally follows these patterns:

0–100 ms Feels instant Around 1 second Delay is noticeable but doesn't interrupt

2–5 seconds Patience starts to wear thin

5–10 seconds Users begin questioning
More than 10 seconds Many users abandon the task altogether

Think about your own browsing habits. If an online store takes ten seconds to load a product page, chances are you'll open another tab and look elsewhere. Users don't compare your application against technical benchmarks. They compare it against their expectations. And expectations keep getting higher.

Performance Through the User's Eyes

One of the biggest mistakes developers make is measuring performance differently from how users experience it.

Imagine a page that displays content after two seconds but remains unresponsive until ten seconds because JavaScript is still executing in the background.

From an engineering perspective, the page loaded.

From the user's perspective, it's broken.

This is why modern web performance focuses on user experience rather than purely technical measurements.

When I evaluate a website's performance, I typically ask four simple questions:

Is something happening?
Is the content useful?
Can I interact with it?
Does everything feel smooth?

These questions map closely to the way users naturally experience a page as it loads.

1. Is Something Happening?
The first thing users want is reassurance. Nobody likes staring at a blank screen. As soon as someone visits a page, they want evidence that the application is working.

Time to First Byte (TTFB)

TTFB measures how long it takes for the browser to receive the first response from the server.

A slow TTFB often points to backend issues such as:

Expensive database queries
Poor caching strategies
Slow APIs
Network latency

@GetMapping("/products")
public List<Product> getProducts() {
    return productRepository.findAll();
}

This looks harmless, but if it triggers multiple database joins or returns thousands of records, response times can increase dramatically. Often, improving performance isn't about adding more servers. It's about removing unnecessary work.
First Paint measures the moment something appears on the screen.

Even if it's just a loading skeleton or navigation bar, users gain confidence because they can see progress. A blank page creates uncertainty. Visible feedback creates trust.

2. Is It Useful?
Showing something quickly is good. Showing meaningful content quickly is better. Users don't visit a website to admire loading indicators. They come for content.

First Contentful Paint (FCP)
FCP measures when meaningful content first appears.
This could be:

Text
Images
Icons
Headlines

At this point, users can begin consuming information.

Largest Contentful Paint (LCP)
LCP measures when the largest visible element appears on screen.In many applications, that's the content users care about most.
Examples include:

A hero image
A product photo
A blog article
A dashboard summary

Google recommends keeping LCP under 2.5 seconds.
A common mistake is loading oversized images:

<img src="huge-image.jpg">

A better approach is to compress assets and use modern formats:

<img
    src="hero.webp"
    width="1200"
    height="600"
>

3. Is It Usable?
Many modern websites look loaded long before they're actually usable.The page appears complete, but clicks don't work.

Buttons feel delayed.
Scrolling becomes sluggish.

This often happens because JavaScript is still busy executing.

Time to Interactive (TTI)
TTI measures when users can reliably interact with the page. A fast-loading page that doesn't respond is still a poor experience.

Interaction to Next Paint (INP)
INP focuses on responsiveness.
It measures how quickly the browser reacts after a user interaction.

For example:

Clicking a button
Opening a menu
Submitting a form

When interactions feel delayed, users notice immediately.
Consider this example:

button.addEventListener("click", () => {
    performMassiveCalculation();
});

If that calculation blocks the main thread for several seconds, the application feels frozen.
A better option could be:

button.addEventListener("click", () => {
    showLoadingIndicator();

    setTimeout(() => {
        performMassiveCalculation();
    }, 0);
});

4. Does It Feel Smooth?
Performance doesn't end once a page loads. Users continue judging your application during scrolling, navigation, and animations.
Small annoyances accumulate quickly.

Cumulative Layout Shift (CLS)
Have you ever tried clicking a button only for the page to suddenly move? That's layout shift.

It's one of the most frustrating performance problems because it interrupts user actions.

A common cause is images loading without reserved dimensions:

<img src="banner.jpg">

A better approach is:

<img
    src="banner.jpg"
    width="1200"
    height="400"
>

Frame Rate
Smooth interfaces generally run at 60 frames per second. That means the browser has roughly 16 milliseconds to render each frame.

Whenever rendering exceeds that budget, users experience:

Stuttering animations
Choppy scrolling
Laggy interactions

The RAIL Model
Google introduced the RAIL model as a way to think about performance from a human perspective rather than a technical one.
RAIL stands for:

Response
Animation
Idle
Load

Each category represents a different part of the user experience.

Response
Users expect feedback almost instantly. If someone clicks a button, they shouldn't wonder whether the click was registered.

A simple loading state often improves perceived performance significantly.

submitButton.onclick = () => {
    showSpinner();
    submitForm();
};

Even when the request takes several seconds, users feel more comfortable because the application is communicating.

Animation
Smooth animations make interfaces feel alive. Poor animations make them feel broken. Whenever possible, animations should remain close to 60 FPS.

For example, avoid expensive calculations during scroll events:

window.addEventListener("scroll", () => {
    expensiveCalculation();
});

Instead, allow the browser to optimize rendering:

window.addEventListener("scroll", () => {
    requestAnimationFrame(updateUI);
});

Idle
Every application has moments when users aren't interacting. These moments are opportunities. Background tasks such as preloading data or caching assets can be performed during idle periods.

requestIdleCallback(() => {
    preloadNextPage();
});

Load
Performance isn't about loading every resource immediately. It's about loading the right resources first.
Techniques such as:

Code splitting
Lazy loading
Server-side rendering
Asset compression
CDN caching

help prioritize what users actually need.

In conclusion Users don't experience your codebase. They experience its speed. They don't see your microservices, design patterns, cloud infrastructure, or database indexes. They see how quickly content appears, how responsive interactions feel, and whether the application helps them accomplish their goals without friction.