DEV Community: Derek mwale

Building a URL Shortener Like a Systems Engineer

Derek mwale — Wed, 22 Jul 2026 17:01:32 +0000

A URL shortener looks like one of the simplest applications you can build.

You paste a long URL.

You get a short link.

Someone clicks it.

They are redirected.

Done.

At least that's how it looks from the outside.

But behind a service like Bitly, TinyURL, or any large-scale link platform is a fascinating systems engineering problem.

A production URL shortener needs to answer questions like:

How do we generate unique short URLs?
How do we handle billions of redirects?
How do we make redirects extremely fast?
How do we prevent abuse?
How do we track analytics?
How do we scale without everything breaking?

The interesting part is not writing the first version.

The interesting part is designing something that survives growth.

Let's build one like a systems engineer.

Understanding the Core Problem

At the surface, the system does two things:

1. Create a Short URL

Input:

https://www.example.com/products/software-development-course

Output:

https://short.ly/a7Kx92

2. Redirect Users

Input:

https://short.ly/a7Kx92

Output:

https://www.example.com/products/software-development-course

The second operation is much more important.

Why?

Because creating links happens occasionally.

Redirects happen constantly.

A popular link might receive:

Thousands of clicks per second
Millions of clicks per day

The system should be optimized for reading, not writing.

High-Level Architecture

A simple architecture might look like this:

                User
                 |
                 |
                 ▼

          API Gateway

                 |
        -----------------
        |               |
        ▼               ▼

 URL Service       Redirect Service

        |               |
        ▼               ▼

   Database          Cache

                         |
                         ▼

                    Analytics

Each component has a clear responsibility.

The URL service creates links.

The redirect service handles traffic.

The cache keeps popular links fast.

Analytics collects information without slowing redirects.

Designing the Database

At the core, we need to store the relationship:

short_code → original_url

A simple table:

CREATE TABLE urls (
    id BIGINT PRIMARY KEY,
    short_code VARCHAR(10) UNIQUE,
    original_url TEXT,
    created_at TIMESTAMP,
    expires_at TIMESTAMP
);

Example:

short_code	original_url
a7Kx92	https://example.com/article
b8Lm21	https://store.com/product

The database answers one important question:

"Where should this short code send the user?"

The Challenge: Generating Short Codes

The biggest design decision is creating the short identifier.

We need something:

Short
Unique
Fast to generate
Collision resistant

A common approach is Base62 encoding.

Base62 uses:

a-z
A-Z
0-9

That's 62 characters.

A code like:

a7Kx92

contains:

62^6

possible combinations.

That gives billions of unique URLs.

Generating IDs

One approach:

Create a unique numeric ID.

Example:

Database ID:

123456

Convert it:

123456 → Base62

→

a7Kx92

The flow:

New URL

    |

Generate ID

    |

Convert ID

    |

Store mapping

    |

Return short URL

Simple.

Reliable.

Fast.

Redirects Are the Critical Path

When someone clicks:

short.ly/a7Kx92

we need the fastest possible response.

The flow:

Request

   |

Extract code

   |

Check Cache

   |

Database Lookup

   |

Redirect User

The first place we look should be memory.

Adding Redis Cache

Most URLs are not equally popular.

A few links receive most traffic.

This is called a:

Long Tail Distribution

Example:

Popular Link

████████████████████

Normal Links

██

Rare Links

█

Instead of hitting the database every time:

User

 |

Redis

 |

Database

The first request loads the URL.

Future requests are served instantly.

Example:

a7Kx92 → https://example.com/article

stored in Redis.

Now redirects happen in milliseconds.

Handling Billions of URLs

A small database works at first.

But eventually:

10 million URLs

100 million URLs

10 billion URLs

becomes a different problem.

We need strategies.

Database Sharding

Instead of one huge database:

Database

   |

----------------

Shard 1

Shard 2

Shard 3

Shard 4

URLs are distributed.

Example:

hash(short_code) % number_of_shards

determines where data lives.

Now traffic is spread across multiple machines.

The Importance of Read Optimization

A URL shortener is a read-heavy system.

Think about the ratio:

Creating URLs:

1 request

Clicking URLs:

1000+ requests

This changes our design priorities.

We optimize:

Caching
Database indexes
Low latency
Fast lookups

Not complicated write systems.

Adding Analytics

Users want to know:

How many people clicked?
Where did they come from?
What devices did they use?
Which countries clicked?

The mistake would be:

User clicks

   |

Save analytics

   |

Redirect

Now analytics slows everything down.

Instead:

User clicks

   |

Redirect immediately

   |

Send event asynchronously

   |

Process analytics later

Using:

Message queues
Event streams
Background workers

keeps the redirect fast.

Preventing Abuse

A URL shortener can easily become an abuse platform.

Attackers may use it for:

Phishing
Spam
Malware distribution

A production system needs:

Rate Limiting

Example:

User A

100 requests/minute

Allowed

User B

10,000 requests/minute

Blocked

URL Scanning

Before creating a link:

Check:

Known malicious domains
Suspicious patterns
Malware databases

Expiration

Some links should not live forever.

Example:

Temporary campaign link

Expires after 30 days

Handling Failures

Distributed systems fail.

Servers crash.

Networks disconnect.

Databases become unavailable.

Design for failure.

Examples:

Database Failure

Use:

Replication
Backups
Failover systems

Cache Failure

The system should still work:

Cache

   X

Database

   ✓

The cache improves speed.

It should never be the only source of truth.

API Design

A simple API:

Create Short URL

POST /api/urls

Request:

{
"url": "https://example.com/article"
}

Response:

{
"short_url": "https://short.ly/a7Kx92"
}

Redirect

GET /a7Kx92

Response:

HTTP 301 Redirect

to:

https://example.com/article

The Technologies I Would Choose

A practical stack:

Backend

Go
Rust
Java
Node.js

Database

PostgreSQL
MySQL
DynamoDB

Cache

Redis

Queue

RabbitMQ
Kafka

Infrastructure

Docker
Kubernetes
Cloud Load Balancer

The technology matters less than the architecture.

What This Project Teaches

A URL shortener is a small project with big lessons.

You learn:

Database design
Indexing
Caching
Distributed systems
Load balancing
Asynchronous processing
Rate limiting
System reliability

This is why system design interviews love URL shorteners.

The application is simple.

The engineering is not.

Final Thoughts

The difference between a beginner implementation and a systems-engineered implementation is not the number of files or the complexity of the code.

It is the ability to think about:

Scale
Failure
Performance
Trade-offs

A simple URL shortener can teach you more about backend architecture than many large projects.

Because great engineers don't just build things that work.

They build things that keep working when millions of people depend on them.

Data Structures That Made My APIs Faster

Derek mwale — Wed, 22 Jul 2026 16:48:43 +0000

When developers talk about making APIs faster, the conversation usually goes straight to:

Adding more servers
Increasing database resources
Adding caching
Optimizing queries
Using faster frameworks

Those things matter.

But sometimes the biggest performance improvement doesn't come from infrastructure.

It comes from choosing the right way to store and access data.

The data structures inside your application quietly decide how fast your API responds.

A bad data structure can turn a millisecond operation into a second-long bottleneck.

A good one can handle millions of operations effortlessly.

Here are the data structures that have had the biggest impact on building faster APIs.

1. Hash Maps: The King of Fast Lookups

One of the most useful data structures in backend development is the hash map.

You probably use it every day without realizing it.

In many languages:

dict

Object

HashMap

HashMap

They all solve the same problem:

"How do I find something quickly?"

Imagine you have an API that retrieves users.

A beginner approach might look like:

users = [
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
    {"id": 3, "name": "Charlie"}
]


def find_user(id):
    for user in users:
        if user["id"] == id:
            return user

This works.

Until you have millions of users.

Every search requires scanning the entire list.

The complexity is:

O(n)

A hash map changes everything.

users = {
    1: {"name": "Alice"},
    2: {"name": "Bob"},
    3: {"name": "Charlie"}
}


user = users[2]

Now the lookup is:

O(1)

Constant time.

The difference between searching one million records and instantly finding one record is enormous.

Real API Example: User Sessions

Imagine an authentication service.

Every request contains:

Authorization: Bearer token123

Your API needs to find the user.

A bad design:

Request
 |
Search all sessions
 |
Find token
 |
Return user

A better design:

Hash Map

token123 → User ID 5421
token456 → User ID 8821
token789 → User ID 9912

Now authentication becomes extremely fast.

This is why caching systems like Redis are built around key-value lookups.

2. Queues: The Backbone of Scalable Systems

Many APIs fail because they try to do everything immediately.

Example:

A user uploads a video.

Your API tries to:

Save the video
Compress it
Generate thumbnails
Send notifications
Update analytics

All before responding.

The user waits.

The server struggles.

A queue changes the architecture.

API Request

      |

      ▼

Add Job To Queue

      |

      ▼

Return Response

      |

      ▼

Background Workers Process Tasks

A queue follows:

First In
First Out

The oldest job gets processed first.

Queues power:

Email systems
Payment processing
Video processing
Notification systems
Data pipelines

A good API doesn't always work harder.

Sometimes it simply moves work somewhere smarter.

3. Trees: Making Search More Efficient

Trees appear everywhere in backend systems.

Databases use tree structures internally.

File systems use trees.

Search engines use trees.

A common example is the B-Tree index.

Imagine this query:

SELECT *
FROM users
WHERE email='derek@example.com';

Without an index:

The database scans every row.

User 1
User 2
User 3
...
User 10,000,000

With an index:

              M
           /     \
        A-M       N-Z
       /             \
   Derek          Sarah

The database doesn't search everything.

It follows a path.

That is the magic of trees.

4. Sets: Eliminating Duplicate Work

Sets are underrated.

They solve a very common backend problem:

"I only want unique values."

Example:

A social media API recommends users to follow.

You collect recommendations from:

Friends
Interests
Location
Trending users

The same person might appear multiple times.

Without a set:

John
Sarah
John
Michael
Sarah

With a set:

John
Sarah
Michael

Instant cleanup.

Sets are useful for:

Removing duplicates
Permission checks
Feature flags
User relationships
Tracking processed events

5. Stacks: The Hidden Power Behind APIs

Stacks follow:

Last In
First Out

The newest item is handled first.

You might not build a stack manually often, but they power many important systems.

Examples:

Function calls
Undo systems
Expression evaluation
Request processing

A practical API example:

Imagine processing nested permissions:

Admin

  └── Manager

        └── Employee

The system can use stack-based traversal to evaluate access rules.

6. Graphs: When Relationships Matter

Modern applications are built on relationships.

Social networks.

Maps.

Recommendations.

Payments.

All of these can be modeled as graphs.

Example:

Derek

 |
 |

Alice ---- Bob

 |
 |

Charlie

A graph contains:

Nodes
Connections

Social media:

User → User

Banking:

Account → Transaction → Account

Travel:

Airport → Flight → Airport

Recommendation engines are basically graph problems.

7. Priority Queues: Handling What Matters First

Not every task has equal importance.

A payment failure notification should probably happen before a marketing email.

Priority queues solve this.

Example:

High Priority

Payment Failed

Password Reset


Medium Priority

Order Update


Low Priority

Newsletter

The system always processes important tasks first.

They power:

Cloud scheduling
Background jobs
Gaming matchmaking
Distributed systems

Data Structures Are Architecture Decisions

A common mistake is thinking:

"Data structures are only for coding interviews."

They are not.

They are architecture decisions.

Every time you build:

An API
A database
A queue system
A search feature
A recommendation engine

you are choosing how information moves.

The choice determines:

Speed
Scalability
Memory usage
Complexity

The Best Backend Engineers Think In Data Structures

A junior developer asks:

"How do I implement this feature?"

An experienced engineer asks:

"What is the right way to represent this problem?"

That difference matters.

A slow API is often not slow because the server is weak.

Sometimes the problem is that the application is searching through lists when it should be using maps.

Or doing synchronous work when it should use queues.

Or scanning everything when it should use indexes.

Final Thoughts

The fastest systems are not always built with the newest technologies.

Sometimes the biggest performance improvements come from understanding the fundamentals.

Hash maps for instant lookups.

Queues for background processing.

Trees for efficient searching.

Sets for uniqueness.

Graphs for relationships.

Priority queues for intelligent scheduling.

Data structures are not just computer science theory.

They are the building blocks behind every fast API, scalable platform, and reliable system we use today.

Before adding more servers, ask a simpler question:

"Am I storing and accessing my data the right way?"

Sometimes the fastest optimization is changing the way you think.

Designing a Notification System That Doesn't Fall Apart

Derek mwale — Wed, 22 Jul 2026 16:35:18 +0000

Every application eventually reaches the point where it needs notifications.

At first, it feels simple.

A user signs up? Send an email.

Someone likes a post? Send a push notification.

An order ships? Send an SMS.

A payment fails? Notify the customer.

So you write something like this:

createOrder();
sendEmail();
sendPushNotification();
sendSMS();

Everything works perfectly.

Until it doesn't.

The first few hundred users never expose the weaknesses in your design. But once your application starts growing, notifications become one of the hardest systems to maintain. Messages are sent twice. Others disappear completely. External providers fail. Users complain they never received important alerts. Suddenly, a feature that seemed trivial becomes responsible for some of the biggest production incidents.

A notification system isn't just about sending messages.

It's about delivering the right message, to the right user, at the right time, through the right channel, reliably.

Here's how I'd build one.

The First Rule: Never Send Notifications Directly

One of the biggest mistakes I see is tightly coupling business logic with notification logic.

For example:

OrderService
    ├── Save Order
    ├── Charge Card
    └── Send Email

Looks harmless.

Now imagine your email provider goes down.

Should customers be prevented from placing orders because an email couldn't be delivered?

Absolutely not.

Notifications should almost never be part of the critical request path.

Instead:

Order Created
        │
        ▼
 Publish Event
        │
        ▼
 Notification Service
        │
        ├── Email
        ├── SMS
        └── Push

The order succeeds immediately.

Notifications happen independently.

That's a much healthier architecture.

Everything Starts With Events

Think in events rather than actions.

Instead of saying:

Send an email.

Say:

OrderCreated happened.

That tiny mindset shift changes everything.

Now multiple systems can react independently.

Send an email.
Send an SMS.
Notify the warehouse.
Update analytics.
Award loyalty points.

Nobody needs to know who is listening.

Your application becomes far easier to extend.

Use Queues Everywhere

External providers are slow.

Some take two seconds.

Some take twenty.

Some randomly fail.

If every request waits for those services, your API becomes painfully slow.

Instead:

API
 │
 ▼
Save Data
 │
 ▼
Queue Notification
 │
 ▼
Return Response

A background worker processes notifications independently.

Benefits include:

Faster APIs
Automatic retries
Better scalability
Failure isolation
Easier monitoring

Queues are one of the simplest improvements you can make to backend reliability.

Every Notification Needs a Lifecycle

Notifications aren't simply "sent."

They move through states.

Pending

Queued

Processing

Sent

Delivered

Failed

Retrying

Expired

Tracking these states helps answer questions like:

Why wasn't this email delivered?
How many SMS messages failed today?
Which notifications are stuck?
How long does delivery usually take?

Without lifecycle tracking, debugging becomes guesswork.

Design Around Multiple Channels

Different users prefer different ways of being notified.

Some want email.

Others want push notifications.

Some businesses require SMS.

Tomorrow you may add Slack, WhatsApp, Microsoft Teams, or Discord.

Avoid hardcoding providers.

Instead:

Notification

Channel Interface

Email

SMS

Push

Slack

Each channel implements the same interface.

Adding a new notification type becomes a matter of plugging in another implementation rather than rewriting existing code.

That's where design patterns like Strategy really shine.

Users Should Control Notifications

Nothing annoys users more than endless alerts.

A notification system should always respect preferences.

For example:

Marketing Emails

Order Updates

Security Alerts

Weekly Reports

Product Announcements

Each category should be independently configurable.

Some notifications should never be disabled, such as password resets or fraud alerts.

Everything else should be optional.

Respecting user preferences builds trust.

Retries Matter More Than You Think

Failures happen.

Networks fail.

SMTP servers fail.

Push providers throttle requests.

SMS gateways become unavailable.

Don't give up after one attempt.

Use exponential backoff.

Instead of retrying immediately:

Retry 1

1 minute

Retry 2

5 minutes

Retry 3

30 minutes

Retry 4

2 hours

This reduces pressure on failing services and dramatically improves delivery success.

Never Lose Notifications

Imagine a payment confirmation disappearing forever because the server restarted.

That's unacceptable.

Always persist notifications before processing.

Database

↓

Queue

↓

Worker

↓

Provider

If the queue crashes, the notification still exists.

If the worker crashes, processing resumes later.

Durability is just as important as speed.

Prevent Duplicate Messages

Retries introduce another challenge.

Duplicate notifications.

Nobody wants five identical emails saying:

"Your payment was successful."

Use idempotency.

Every notification should have a unique identifier.

Before sending, check whether that notification has already been successfully delivered.

Processing the same event twice should still produce only one user-facing message.

Separate Templates From Code

Hardcoding notification text quickly becomes painful.

Instead of this:

sendEmail("Hello John, your order has shipped...")

Use templates.

Template:
Order Shipped

Variables:
Customer Name
Tracking Number
Courier

This allows designers, marketers, or support teams to update messaging without touching application code.

It's also essential if your application supports multiple languages.

Monitor Everything

A notification system without observability is impossible to operate.

Track metrics such as:

Delivery rate
Failure rate
Retry count
Queue size
Processing time
Provider latency
Bounce rate
Open rate (where applicable)

Good dashboards often reveal problems long before users notice them.

Scaling Is Easier Than You Think

One beautiful aspect of queue-based systems is horizontal scaling.

One worker processes 100 notifications per minute.

Need more?

Run five workers.

Need even more?

Run fifty.

No changes to your application code.

That's one of the biggest reasons modern distributed systems rely so heavily on message queues.

A Simple Architecture

Putting everything together, the architecture looks something like this:

Application
        │
        ▼
Event Bus
        │
        ▼
Notification Queue
        │
        ▼
Notification Workers
        │
        ▼
Email | SMS | Push | Slack
        │
        ▼
Delivery Status Database

Each component has a single responsibility.

The application creates events.

The queue guarantees reliable processing.

Workers send notifications.

Providers deliver messages.

The database tracks everything.

Each layer can evolve independently.

Final Thoughts

Notifications look like a small feature until your product starts growing.

Then they become one of the busiest systems in your entire platform.

The difference between a notification system that scales and one that constantly breaks usually isn't a fancy technology or a complex algorithm. It's thoughtful architecture.

Decouple your business logic.

Think in events.

Use queues.

Design for retries.

Make operations idempotent.

Respect user preferences.

Measure everything.

Build it this way from the beginning, and your notification system won't just send messages—it will keep doing its job reliably, even as your application grows from hundreds of users to millions.