DEV Community: Hardik Agarwal

Designing a URL Shortener: My First High-Level System Design

Hardik Agarwal — Sat, 24 Oct 2026 23:03:08 +0000

Exploring API design, Redis caching, database capacity, horizontal scaling, and asynchronous analytics.

System design has always seemed like one of those skills that becomes easier only when you start building systems on paper.

So, I decided to start practicing High-Level Design (HLD) by designing a familiar application: a URL shortener.

At first glance, a URL shortener looks simple. You provide a long URL, receive a short link, and redirect users when they visit it.

But what happens when millions of users create links and billions of redirects need to be processed? How do we keep redirects fast? How do we handle traffic spikes? And how do we collect analytics without slowing down the redirect path?

In this article, I'll walk through my first-pass design, the reasoning behind the architecture, and the trade-offs I need to understand better.

1. Understanding the requirements

Before drawing boxes and connecting services, I started by defining what the system needs to do.

Functional requirements

The initial scope includes two core features:

Create a short code for an original URL.
Redirect a short URL to its original destination.

Click analytics are also included as an asynchronous supporting feature.

Non-functional requirements

The system should aim for:

High availability.
Low redirect latency.
Horizontal scalability.
Durable URL mappings.
Asynchronous analytics processing.

For this exercise, I set illustrative latency targets:

Operation	p95 target	p99 target
Create URL	600 ms	800 ms
Redirect	100 ms	200 ms

These are design targets, not measured results. Actual latency depends on deployment region, network conditions, cache performance, and infrastructure.

I also assumed that URLs expire after seven days by default for this initial design.

2. The proposed architecture

The architecture contains the following components:

Client
NGINX / load balancer
Horizontally scaled URL service
Redis cache
Primary database
Message queue
Analytics worker and analytics database
Prometheus and Grafana
A scheduled cleanup job

The main idea is to keep the redirect path fast while moving work that doesn't need to happen immediately into background processing.

The request flow looks like this:

URL creation

Client → Load Balancer → URL Service → Primary Database → Return Short URL

URL redirection

Client → Load Balancer → URL Service → Redis Cache → Redirect

If the cache misses, the URL service reads from the primary database and can populate the cache for subsequent requests.

Click analytics

URL Service → Message Queue → Analytics Worker → Analytics Database

Monitoring

URL Service and supporting components → Prometheus → Grafana

Each component solves a different problem, and understanding why it exists is more important than simply memorizing the architecture.

3. Why do we need a load balancer?

A single application server can become a bottleneck as traffic grows.

Instead of directing every request to one server, the load balancer distributes requests across multiple instances of the URL service.

For example, we might initially run three application instances.

This gives us several advantages:

We can add instances as traffic increases.
A failed instance can be removed from the available pool.
Requests can be distributed across healthy instances.
We can deploy updated instances without necessarily taking the entire service offline.

The load balancer doesn't make the whole system highly available by itself. The application instances, database, cache, and other dependencies must also be designed to tolerate failures.

4. Why Redis is important

This was one of the most interesting parts of the design for me.

A URL shortener receives many redirect requests, and some short links may be accessed far more often than others.

If every redirect required a database query, the primary database would handle a large amount of repetitive work.

Redis provides a fast in-memory lookup layer for frequently accessed URL mappings.

A simplified cache entry might look like this:

short_code → original_url

For example:

aB91xZ → https://example.com/articles/system-design

Cache hit

The URL service receives a redirect request.
It checks Redis using the short code.
Redis returns the original URL.
The service responds with an HTTP redirect.

Cache miss

Redis doesn't contain the requested mapping.
The URL service queries the primary database.
If the mapping exists and is valid, the service returns the redirect and populates the cache.
Future requests may be served directly from Redis.

Expired or missing URLs should not be treated as valid destinations.

In a production system, I would also consider cache stampedes, negative caching, eviction policies, and what happens when Redis becomes unavailable.

The key lesson: Redis is a performance optimization, while the primary database remains the durable source of URL mappings.

5. Choosing the primary database

The primary database stores the authoritative URL mapping.

A simplified schema could look like this:

Column	Purpose
`id`	Primary key
`original_url`	Original destination
`short_code`	Unique, indexed short code
`created_at`	Creation timestamp
`expires_at`	Expiration timestamp

The short_code must be unique. Otherwise, two different URLs could end up sharing the same short link.

If a generated code collides with an existing code, the service needs to retry or use another collision-resolution strategy.

I would also index short_code because redirect requests use it for lookups.

The database choice depends on the workload and consistency requirements. A relational database is a reasonable starting point because it supports uniqueness constraints and durable storage. At larger scales, partitioning, replication, and the distribution of data would need further investigation.

6. Estimating traffic before choosing infrastructure

Capacity estimation is where system design becomes more than drawing architecture diagrams.

For this exercise, I used the following assumptions:

Metric	Assumption
Total registered users	100 million
Daily active users	10 million
URL creations per active user per day	5
Redirects per creation	100
Seconds per day	86,400

These are illustrative assumptions rather than actual usage statistics.

URL creation requests

Daily creations:

10 million × 5 = 50 million creations/day.

Average creation requests per second:

50,000,000 ÷ 86,400 ≈ 579 requests/second.

Redirect requests

Assuming a 100:1 redirect-to-creation ratio:

50 million × 100 = 5 billion redirects/day.

Average redirect requests per second:

5,000,000,000 ÷ 86,400 ≈ 57,870 requests/second.

That means the system needs to handle substantially more redirects than creation requests.

This is an important design insight: the redirect path deserves particular attention because it carries most of the traffic in this workload.

Average traffic is not enough for infrastructure sizing. If peak traffic reaches ten times the average, the redirect path could approach 579,000 requests per second under these assumptions.

That would require additional capacity planning, load testing, and careful consideration of cache efficiency and database bottlenecks.

7. Estimating database storage

Next, I estimated how much space URL mappings might occupy.

Suppose the average record and its relevant index overhead consume approximately 138 bytes. This is a simplified planning estimate, not a measured database row size.

For 500 million mappings:

500,000,000 × 138 bytes ≈ 69 GB.

This is a rough estimate for the assumed per-entry footprint. Actual storage will vary based on URL lengths, index structures, database format, replication, metadata, and storage overhead.

A production capacity estimate should also account for backups, replicas, growth, and operational headroom.

The takeaway is that storage calculations should begin with explicit assumptions and be refined using measurements from the actual database.

8. Why asynchronous analytics?

A click event is useful for understanding how often links are accessed and how traffic changes over time.

However, I don't want the redirect request to wait for analytics aggregation or database writes.

Instead, the URL service publishes a click event to a message queue, and an analytics worker processes it independently.

The worker can aggregate information and write it to a separate analytics database.

This separation provides several benefits:

Analytics processing doesn't need to run synchronously with each redirect.
Workers can scale independently.
Events can be retried after transient processing failures.
Analytics storage can be optimized for reporting rather than URL lookups.

There are trade-offs, too. Analytics may be delayed, queues require monitoring, and retries can produce duplicate events unless processing is designed to be idempotent.

The exact delivery guarantees depend on the queue and consumer implementation.

For this first design, the primary database is responsible for URL mappings, while the analytics database is responsible for click statistics.

9. Expiration and background cleanup

The initial requirement gives each URL a default lifetime of seven days.

A scheduled job can periodically identify expired mappings and remove them from the primary database.

But deleting a record is only one part of expiration handling.

The service must also ensure that expired URLs aren't served from Redis. Otherwise, a stale cache entry could continue redirecting users after the database record has expired.

Possible approaches include cache TTLs aligned with URL expiration, expiration checks during lookup, and invalidation when records are deleted.

For this exercise, I placed a cleanup job in the architecture that runs every three hours. That interval is a starting assumption, not a guarantee that every expired mapping disappears immediately.

10. Observability with Prometheus and Grafana

A system can appear healthy while users experience slow redirects or requests begin failing.

I included Prometheus and Grafana to make the system's behavior visible.

Some useful metrics include:

Request rate and error rate for the URL service.
Redirect latency at p50, p95, and p99.
Redis hit ratio and latency.
Primary database query latency and connection utilization.
Queue depth and event-processing delay.
Analytics worker failures and retry counts.
CPU, memory, and instance health.

Prometheus collects metrics, while Grafana visualizes them through dashboards and alerts.

I would also add structured logs and distributed tracing where appropriate. Metrics tell us that something is wrong; logs and traces help us investigate why.

11. Trade-offs and questions for the next iteration

This is a first-pass architecture, not a finished production design. Several questions still need deeper analysis.

How should short codes be generated?

Random codes, counter-based IDs, and IDs encoded using Base62 have different trade-offs involving collisions, predictability, coordination, and distribution.

What happens when Redis fails?

The service may need to fall back to the database, but an uncontrolled fallback could overwhelm it. Timeouts, circuit breakers, concurrency limits, and graceful degradation would need consideration.

How do we handle very popular URLs?

A small number of links could account for a large fraction of traffic. Cache strategy, replication, and hot-key handling become important.

How do we guarantee reliable analytics events?

Publishing events introduces failure scenarios between serving a redirect and recording its click. A more detailed design should establish the required delivery guarantees and consider durable event publication patterns.

When should we partition the database?

That decision should follow evidence about storage growth, query performance, write throughput, and operational constraints rather than being introduced automatically.

These are the areas I want to explore in future iterations.

Final thoughts

Designing a URL shortener helped me connect several backend concepts that are often learned separately: caching, database indexing, load balancing, asynchronous processing, capacity estimation, and observability.

The biggest lesson was that a system design isn't just a collection of technologies. Every component should have a clear purpose, and every major decision should have an explanation and a trade-off.

This is my first iteration, and there is plenty left to improve.

I'm continuing to practice system design by choosing familiar products, defining requirements, estimating scale, drawing architectures, and investigating the trade-offs.

I'd love feedback from engineers and fellow learners: What would you change first in this design, and why?

SystemDesign #BackendEngineering #SoftwareArchitecture #DistributedSystems #LearningInPublic

npm run dev vs npm start: The Difference That Breaks Your App in Production

Hardik Agarwal — Mon, 30 Mar 2026 18:12:07 +0000

Everything was working perfectly.

I ran:

npm run dev

And my app felt amazing.

It reloaded instantly
It never stayed broken
Errors? Fixed themselves after save

Honestly, it felt like I had built something solid.

Then I deployed it.

And everything fell apart.

What Went Wrong?

The app started crashing.

Sometimes it just stopped responding.
Sometimes logs didn’t show anything useful.
And the worst part?

It didn’t recover on its own.

I kept thinking:

“But it was working perfectly on my machine…”

The Tiny Difference That Caused All This

It came down to one small change:

"dev": "nodemon index.js",
"start": "node index.js"

That’s it.

Just:

nodemon in development
node in production

At first, this looks harmless.

But it changes everything.

What nodemon Actually Does (And Why It Tricks You)

When you use nodemon, your app behaves very differently:

It watches your files
It restarts automatically on every change
It even restarts after crashes

So in development, your app feels… immortal.

You break something? Save → fixed.
Server crashes? It comes back instantly.

You never really see failure.

But Production Is Not That Forgiving

When you run:

node index.js

Things are very different.

No auto restart
No file watching
No safety net

If your app crashes…

👉 it stays down.

No warning. No recovery. Just silence.

The Mistake Most Developers Make

We get used to this loop:

Code → Save → Auto restart → Everything works

And we assume:

“Yeah, this app is stable.”

But the truth is:

👉 nodemon was hiding your problems

Real Issues That Only Show Up in Production

1. Crashes are permanent

In dev:

Crash → nodemon restarts → you move on

In prod:

Crash → your server is dead

2. You don’t have a process manager

Running this in production:

node index.js

is risky.

If it crashes, nobody brings it back.

That’s why real systems use tools like PM2.

They:

Restart your app automatically
Keep logs
Handle multiple instances

3. Your environment is different (and you forgot)

In development, everything is relaxed.

In production, you should be running:

NODE_ENV=production node index.js

But many people forget this.

And suddenly:

Debug logs flood your system
Performance drops
Sensitive info might leak

4. Logging suddenly becomes useless

In dev:

console.log("works great")

In prod:

Logs are messy or missing
Debugging becomes painful

You realize:

“console.log is not a logging strategy.”

5. You never tested real conditions

Your app worked with:

1 user
0 pressure
perfect conditions

But production?

100+ users
concurrent requests
unpredictable failures

Tools like Autocannon exist for a reason.

Because real systems behave very differently under load.

The Real Lesson

The problem is not nodemon.

The problem is this assumption:

“If it works in development, it will work in production.”

That’s not true.

What You Should Actually Do

Here’s a better setup:

"scripts": {
  "dev": "nodemon index.js",
  "start": "node index.js",
  "prod": "pm2 start index.js"
}

And more importantly:

Use nodemon only for development
Always test using node before deploying
Use a process manager in production
Set NODE_ENV=production
Don’t rely on auto-restarts

One Line That Changed My Thinking

After going through all this, I realized:

👉 npm run dev helps you build fast
👉 npm start decides if your app survives

Final Thought

Development is comfortable.

It fixes your mistakes.
It hides your crashes.
It makes everything feel smooth.

Production does the opposite.

It exposes everything.

So next time your app works perfectly on your machine…

Pause for a second and ask:

“Would this still work… without nodemon watching my back?”

That question alone can save you hours of debugging.

—and maybe even your production system.