Exploring API design, Redis caching, database capacity, horizontal scaling, and asynchronous analytics.
System design has always seemed like one of those skills that becomes easier only when you start building systems on paper.
So, I decided to start practicing High-Level Design (HLD) by designing a familiar application: a URL shortener.
At first glance, a URL shortener looks simple. You provide a long URL, receive a short link, and redirect users when they visit it.
But what happens when millions of users create links and billions of redirects need to be processed? How do we keep redirects fast? How do we handle traffic spikes? And how do we collect analytics without slowing down the redirect path?
In this article, I'll walk through my first-pass design, the reasoning behind the architecture, and the trade-offs I need to understand better.
1. Understanding the requirements
Before drawing boxes and connecting services, I started by defining what the system needs to do.
Functional requirements
The initial scope includes two core features:
- Create a short code for an original URL.
- Redirect a short URL to its original destination.
Click analytics are also included as an asynchronous supporting feature.
Non-functional requirements
The system should aim for:
- High availability.
- Low redirect latency.
- Horizontal scalability.
- Durable URL mappings.
- Asynchronous analytics processing.
For this exercise, I set illustrative latency targets:
| Operation | p95 target | p99 target |
|---|---|---|
| Create URL | 600 ms | 800 ms |
| Redirect | 100 ms | 200 ms |
These are design targets, not measured results. Actual latency depends on deployment region, network conditions, cache performance, and infrastructure.
I also assumed that URLs expire after seven days by default for this initial design.
2. The proposed architecture
The architecture contains the following components:
- Client
- NGINX / load balancer
- Horizontally scaled URL service
- Redis cache
- Primary database
- Message queue
- Analytics worker and analytics database
- Prometheus and Grafana
- A scheduled cleanup job
The main idea is to keep the redirect path fast while moving work that doesn't need to happen immediately into background processing.
The request flow looks like this:
URL creation
Client → Load Balancer → URL Service → Primary Database → Return Short URL
URL redirection
Client → Load Balancer → URL Service → Redis Cache → Redirect
If the cache misses, the URL service reads from the primary database and can populate the cache for subsequent requests.
Click analytics
URL Service → Message Queue → Analytics Worker → Analytics Database
Monitoring
URL Service and supporting components → Prometheus → Grafana
Each component solves a different problem, and understanding why it exists is more important than simply memorizing the architecture.
3. Why do we need a load balancer?
A single application server can become a bottleneck as traffic grows.
Instead of directing every request to one server, the load balancer distributes requests across multiple instances of the URL service.
For example, we might initially run three application instances.
This gives us several advantages:
- We can add instances as traffic increases.
- A failed instance can be removed from the available pool.
- Requests can be distributed across healthy instances.
- We can deploy updated instances without necessarily taking the entire service offline.
The load balancer doesn't make the whole system highly available by itself. The application instances, database, cache, and other dependencies must also be designed to tolerate failures.
4. Why Redis is important
This was one of the most interesting parts of the design for me.
A URL shortener receives many redirect requests, and some short links may be accessed far more often than others.
If every redirect required a database query, the primary database would handle a large amount of repetitive work.
Redis provides a fast in-memory lookup layer for frequently accessed URL mappings.
A simplified cache entry might look like this:
short_code → original_url
For example:
aB91xZ → https://example.com/articles/system-design
Cache hit
- The URL service receives a redirect request.
- It checks Redis using the short code.
- Redis returns the original URL.
- The service responds with an HTTP redirect.
Cache miss
- Redis doesn't contain the requested mapping.
- The URL service queries the primary database.
- If the mapping exists and is valid, the service returns the redirect and populates the cache.
- Future requests may be served directly from Redis.
Expired or missing URLs should not be treated as valid destinations.
In a production system, I would also consider cache stampedes, negative caching, eviction policies, and what happens when Redis becomes unavailable.
The key lesson: Redis is a performance optimization, while the primary database remains the durable source of URL mappings.
5. Choosing the primary database
The primary database stores the authoritative URL mapping.
A simplified schema could look like this:
| Column | Purpose |
|---|---|
id |
Primary key |
original_url |
Original destination |
short_code |
Unique, indexed short code |
created_at |
Creation timestamp |
expires_at |
Expiration timestamp |
The short_code must be unique. Otherwise, two different URLs could end up sharing the same short link.
If a generated code collides with an existing code, the service needs to retry or use another collision-resolution strategy.
I would also index short_code because redirect requests use it for lookups.
The database choice depends on the workload and consistency requirements. A relational database is a reasonable starting point because it supports uniqueness constraints and durable storage. At larger scales, partitioning, replication, and the distribution of data would need further investigation.
6. Estimating traffic before choosing infrastructure
Capacity estimation is where system design becomes more than drawing architecture diagrams.
For this exercise, I used the following assumptions:
| Metric | Assumption |
|---|---|
| Total registered users | 100 million |
| Daily active users | 10 million |
| URL creations per active user per day | 5 |
| Redirects per creation | 100 |
| Seconds per day | 86,400 |
These are illustrative assumptions rather than actual usage statistics.
URL creation requests
Daily creations:
10 million × 5 = 50 million creations/day.
Average creation requests per second:
50,000,000 ÷ 86,400 ≈ 579 requests/second.
Redirect requests
Assuming a 100:1 redirect-to-creation ratio:
50 million × 100 = 5 billion redirects/day.
Average redirect requests per second:
5,000,000,000 ÷ 86,400 ≈ 57,870 requests/second.
That means the system needs to handle substantially more redirects than creation requests.
This is an important design insight: the redirect path deserves particular attention because it carries most of the traffic in this workload.
Average traffic is not enough for infrastructure sizing. If peak traffic reaches ten times the average, the redirect path could approach 579,000 requests per second under these assumptions.
That would require additional capacity planning, load testing, and careful consideration of cache efficiency and database bottlenecks.
7. Estimating database storage
Next, I estimated how much space URL mappings might occupy.
Suppose the average record and its relevant index overhead consume approximately 138 bytes. This is a simplified planning estimate, not a measured database row size.
For 500 million mappings:
500,000,000 × 138 bytes ≈ 69 GB.
This is a rough estimate for the assumed per-entry footprint. Actual storage will vary based on URL lengths, index structures, database format, replication, metadata, and storage overhead.
A production capacity estimate should also account for backups, replicas, growth, and operational headroom.
The takeaway is that storage calculations should begin with explicit assumptions and be refined using measurements from the actual database.
8. Why asynchronous analytics?
A click event is useful for understanding how often links are accessed and how traffic changes over time.
However, I don't want the redirect request to wait for analytics aggregation or database writes.
Instead, the URL service publishes a click event to a message queue, and an analytics worker processes it independently.
The worker can aggregate information and write it to a separate analytics database.
This separation provides several benefits:
- Analytics processing doesn't need to run synchronously with each redirect.
- Workers can scale independently.
- Events can be retried after transient processing failures.
- Analytics storage can be optimized for reporting rather than URL lookups.
There are trade-offs, too. Analytics may be delayed, queues require monitoring, and retries can produce duplicate events unless processing is designed to be idempotent.
The exact delivery guarantees depend on the queue and consumer implementation.
For this first design, the primary database is responsible for URL mappings, while the analytics database is responsible for click statistics.
9. Expiration and background cleanup
The initial requirement gives each URL a default lifetime of seven days.
A scheduled job can periodically identify expired mappings and remove them from the primary database.
But deleting a record is only one part of expiration handling.
The service must also ensure that expired URLs aren't served from Redis. Otherwise, a stale cache entry could continue redirecting users after the database record has expired.
Possible approaches include cache TTLs aligned with URL expiration, expiration checks during lookup, and invalidation when records are deleted.
For this exercise, I placed a cleanup job in the architecture that runs every three hours. That interval is a starting assumption, not a guarantee that every expired mapping disappears immediately.
10. Observability with Prometheus and Grafana
A system can appear healthy while users experience slow redirects or requests begin failing.
I included Prometheus and Grafana to make the system's behavior visible.
Some useful metrics include:
- Request rate and error rate for the URL service.
- Redirect latency at p50, p95, and p99.
- Redis hit ratio and latency.
- Primary database query latency and connection utilization.
- Queue depth and event-processing delay.
- Analytics worker failures and retry counts.
- CPU, memory, and instance health.
Prometheus collects metrics, while Grafana visualizes them through dashboards and alerts.
I would also add structured logs and distributed tracing where appropriate. Metrics tell us that something is wrong; logs and traces help us investigate why.
11. Trade-offs and questions for the next iteration
This is a first-pass architecture, not a finished production design. Several questions still need deeper analysis.
How should short codes be generated?
Random codes, counter-based IDs, and IDs encoded using Base62 have different trade-offs involving collisions, predictability, coordination, and distribution.
What happens when Redis fails?
The service may need to fall back to the database, but an uncontrolled fallback could overwhelm it. Timeouts, circuit breakers, concurrency limits, and graceful degradation would need consideration.
How do we handle very popular URLs?
A small number of links could account for a large fraction of traffic. Cache strategy, replication, and hot-key handling become important.
How do we guarantee reliable analytics events?
Publishing events introduces failure scenarios between serving a redirect and recording its click. A more detailed design should establish the required delivery guarantees and consider durable event publication patterns.
When should we partition the database?
That decision should follow evidence about storage growth, query performance, write throughput, and operational constraints rather than being introduced automatically.
These are the areas I want to explore in future iterations.
Final thoughts
Designing a URL shortener helped me connect several backend concepts that are often learned separately: caching, database indexing, load balancing, asynchronous processing, capacity estimation, and observability.
The biggest lesson was that a system design isn't just a collection of technologies. Every component should have a clear purpose, and every major decision should have an explanation and a trade-off.
This is my first iteration, and there is plenty left to improve.
I'm continuing to practice system design by choosing familiar products, defining requirements, estimating scale, drawing architectures, and investigating the trade-offs.
I'd love feedback from engineers and fellow learners: What would you change first in this design, and why?
Top comments (0)