DEV Community

chaitanya emani
chaitanya emani

Posted on

Designing Pastebin: What I Learned About System Design

#systemdesign #webdev #backend #mongodb

After designing a URL Shortener, I wanted to work on another system-design problem that would expose me to different concepts.

So I picked Pastebin.

At first, I thought the problem would be simple:

Store some text and return a URL.

But once I started thinking about scale, expiration, large content, caching, database selection, and failure scenarios, there was much more to it.

More importantly, I wanted to change how I approached system design.

Instead of starting with:

"Which technologies should I use?"

I started with:

"What problem am I actually trying to solve?"

This article walks through my design and, more importantly, the reasoning behind it.

1. What are we building?

The basic idea is simple.

A user submits some text:

Hello, this is my paste!
Enter fullscreen mode Exit fullscreen mode

The system generates a unique URL:

https://pastebin.com/abc123
Enter fullscreen mode Exit fullscreen mode

Anyone with the URL can retrieve the paste.

For this design, I considered these requirements:

Functional requirements

  • Create a paste
  • Retrieve a paste
  • Delete a paste
  • Set an expiration time
  • No authentication
  • No editing after creation

Since there is no authentication, deletion is handled using a delete token generated when the paste is created.

2. What happens when a paste is created?

The basic flow looks like this:

Client
   |
   v
POST /pastes
   |
   v
API Server
   |
   +---- Generate Snowflake ID
   |
   +---- Convert ID to Base62
   |
   +---- Generate Delete Token
   |
   +---- Check content size
              |
        +-----+-----+
        |           |
      <= 1 MB      > 1 MB
        |           |
        v           v
    MongoDB         S3
        |           |
        +-----+-----+
              |
              v
        Return paste URL
Enter fullscreen mode Exit fullscreen mode

The interesting part here is the storage decision.

3. MongoDB or S3?

This was one of the biggest decisions I had to think about.

Initially, I considered storing everything in MongoDB.

But then I asked:

What happens when paste contents become large?

A database is not necessarily the best place for every type of data.

So I separated metadata from large objects.

MongoDB

MongoDB stores:

pasteId
deleteTokenHash
storageType
objectKey
createdAt
expiresAt
content
Enter fullscreen mode Exit fullscreen mode

S3

Amazon S3 stores the actual content when the paste is large.

My application-level rule is:

Content <= 1 MB
        ↓
    MongoDB

Content > 1 MB
        ↓
       S3
Enter fullscreen mode Exit fullscreen mode

So MongoDB remains responsible for application metadata, while S3 handles larger paste content.

4. Why MongoDB?

One of the things I realized while working on Pastebin was that database selection should start from requirements and access patterns.

Pastebin doesn't have complex relationships between entities.

A paste is mostly an independent document:

Paste
 ├── pasteId
 ├── deleteToken
 ├── createdAt
 ├── expiresAt
 └── content / objectKey
Enter fullscreen mode Exit fullscreen mode

So a document database such as MongoDB is a reasonable choice for this design.

The important lesson for me wasn't:

"Pastebin uses MongoDB."

It was:

Understand the data and access patterns first, then choose the database.

5. Generating the Paste ID

I reused an idea from my URL Shortener design:

Snowflake ID + Base62

The process is:

Snowflake
    ↓
Unique numeric ID
    ↓
Base62 encoding
    ↓
Short paste ID
Enter fullscreen mode Exit fullscreen mode

For example:

123456789012345
        ↓
     Base62
        ↓
     8dK92x
Enter fullscreen mode Exit fullscreen mode

The final URL becomes:

https://pastebin.com/8dK92x
Enter fullscreen mode Exit fullscreen mode

Here, Snowflake solves the distributed ID generation problem, while Base62 makes the representation shorter.

6. Database Schema

The final MongoDB document looks approximately like this:

{
    pasteId: "8dK92x",

    deleteTokenHash: "...",

    storageType: "DB",

    content: "Hello, World!",

    objectKey: null,

    createdAt: "...",

    expiresAt: "..."
}
Enter fullscreen mode Exit fullscreen mode

For a large paste:

{
    pasteId: "7xP92a",

    deleteTokenHash: "...",

    storageType: "S3",

    content: null,

    objectKey: "pastes/7xP92a",

    createdAt: "...",

    expiresAt: "..."
}
Enter fullscreen mode Exit fullscreen mode

This gives us a clear separation:

Small content
    → MongoDB

Large content
    → S3
Enter fullscreen mode Exit fullscreen mode

7. API Design

The APIs are intentionally simple.

Create paste 

POST /pastes
Enter fullscreen mode Exit fullscreen mode

Request:

{
    "content": "Hello World",
    "expiration": "1d"
}
Enter fullscreen mode Exit fullscreen mode

Response:

{
    "pasteId": "8dK92x",
    "url": "https://pastebin.com/8dK92x",
    "deleteToken": "..."
}
Enter fullscreen mode Exit fullscreen mode

Retrieve paste 

GET /pastes/:pasteId
Enter fullscreen mode Exit fullscreen mode

The server checks MongoDB first.

If:

storageType = DB
Enter fullscreen mode Exit fullscreen mode

return the content directly.

If:

storageType = S3
Enter fullscreen mode Exit fullscreen mode

use objectKey to retrieve the content from S3.

Delete paste 

DELETE /pastes/:pasteId
Enter fullscreen mode Exit fullscreen mode

The delete token can be provided through a request header:

X-Delete-Token: <token>
Enter fullscreen mode Exit fullscreen mode

The server verifies the token before deleting the paste.

8. What happens when a paste expires?

This was another interesting part of the design.

Suppose:

expiresAt = 10:00 AM
Enter fullscreen mode Exit fullscreen mode

At 10:05 AM, somebody requests the paste.

We shouldn't depend entirely on a background cleanup process.

The API checks:

expiresAt < currentTime
Enter fullscreen mode Exit fullscreen mode

If true:

Return 404 / Expired
Enter fullscreen mode Exit fullscreen mode

The paste is therefore logically expired immediately.

But the physical data may still exist temporarily in:

  • MongoDB
  • S3
  • Redis
  • CDN

That's where background workers come in.

9. Background Workers

A background worker periodically finds expired pastes.

             Background Worker
                    |
             Find expired pastes
                    |
          +---------+---------+
          |         |         |
          v         v         v
       MongoDB     S3       Cache
       delete     delete    invalidate
Enter fullscreen mode Exit fullscreen mode

For example:

Paste expires
     ↓
Worker detects expiration
     ↓
Delete MongoDB metadata
     ↓
Delete S3 object if present
     ↓
Invalidate cached data
Enter fullscreen mode Exit fullscreen mode

This taught me an important distinction:

Expiration and deletion are not necessarily the same operation.

The application can immediately treat something as expired while asynchronous processes handle physical cleanup.

10. Caching

Pastebin is naturally read-heavy.

Suppose we have:

10M writes/day
100M reads/day
Enter fullscreen mode Exit fullscreen mode

Repeatedly querying the database for the same popular paste would create unnecessary load.

So I introduced caching.

L1 Cache

Each API server has a local in-memory cache.

API Server
    ↓
L1 Cache
Enter fullscreen mode Exit fullscreen mode

L2 Cache

If L1 misses:

L1 Cache
    ↓ miss
Redis
    ↓ miss
MongoDB / S3
Enter fullscreen mode Exit fullscreen mode

This gives us:

L1 → fastest
L2 → shared cache
DB/S3 → source of truth
Enter fullscreen mode Exit fullscreen mode

11. What if Redis goes down?

This was one of the questions I found particularly useful while designing the system.

If Redis is unavailable:

L1
 ↓ miss
Redis
 ↓ unavailable
MongoDB
Enter fullscreen mode Exit fullscreen mode

The system becomes slower, but the data isn't lost.

This reinforced an important principle:

A cache should improve performance, not become the only source of truth.

Redis can also be deployed with high availability when the workload requires it.

12. CDN

What if a paste suddenly becomes extremely popular?

For example:

1 paste
   ↓
millions of requests
Enter fullscreen mode Exit fullscreen mode

Sending every request to our API servers would be wasteful.

A CDN can cache frequently accessed content closer to users.

User
  ↓
 CDN
  ↓ cache hit
Paste Content
Enter fullscreen mode Exit fullscreen mode

This allows the CDN to absorb a large portion of read traffic before requests reach our backend.

13. Load Balancer

We don't want a single API server handling everything.

Instead:

             Load Balancer
             /     |     \
            /      |      \
         API 1   API 2   API 3
Enter fullscreen mode Exit fullscreen mode

The API servers are stateless, so requests can be distributed across them.

If traffic increases:

3 servers
    ↓
10 servers
    ↓
50 servers
Enter fullscreen mode Exit fullscreen mode

This gives us horizontal scalability.

14. Database Scaling

As the system grows, a single MongoDB instance may eventually become a bottleneck.

Replication

We can have:

             Primary
            /       \
           /         \
      Replica 1    Replica 2
Enter fullscreen mode Exit fullscreen mode

Replication improves availability and can also support read scaling depending on consistency requirements.

Sharding

If the dataset or traffic becomes too large:

             MongoDB
                |
       +--------+--------+
       |        |        |
     Shard 1  Shard 2  Shard 3
Enter fullscreen mode Exit fullscreen mode

We can distribute data across multiple shards using an appropriate shard key.

15. Rate Limiting

Pastebin is a public service, so we also need to protect it from abuse.

For example:

100 requests / minute / IP
Enter fullscreen mode Exit fullscreen mode

A rate limiter can sit before the application logic:

User
 ↓
Rate Limiter
 ↓
Load Balancer
 ↓
API Servers
Enter fullscreen mode Exit fullscreen mode

Redis can be used to maintain distributed rate-limit counters.

16. Final Architecture

Putting everything together:

                         USERS
                           |
                           v
                          CDN
                           |
                      Rate Limiter
                           |
                           v
                    Load Balancer
                           |
             +-------------+-------------+
             |             |             |
             v             v             v
          API #1        API #2        API #3
             |             |             |
             +-------------+-------------+
                           |
                           v
                       L1 Cache
                           |
                        Cache Miss
                           |
                           v
                      Redis (L2)
                           |
                        Cache Miss
                           |
                           v
                    +-------------+
                    |  MongoDB    |
                    |             |
                    | Metadata    |
                    | Small Data  |
                    +------+------+
                           |
                     Large Content
                           |
                           v
                         S3


                 Background Workers
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
          MongoDB       S3        Cache
          Cleanup     Cleanup    Invalidation
Enter fullscreen mode Exit fullscreen mode

17. What I Learned

The biggest lesson from this project wasn't MongoDB, Redis, S3, or CDN.

It was how to think about system design.

Initially, I was approaching system design like this:

"I know Redis.
 I know MongoDB.
 I know CDN.
 Let me add them to the architecture."
Enter fullscreen mode Exit fullscreen mode

Now I'm trying to approach it like this:

Requirements
     ↓
Traffic & Workload
     ↓
Data & Access Patterns
     ↓
Identify Bottlenecks
     ↓
Consider Alternatives
     ↓
Choose Components
     ↓
Understand Trade-offs
     ↓
Design Architecture
Enter fullscreen mode Exit fullscreen mode

For example:

Why S3?

Not simply because:

"S3 is popular."

But because:

"Large paste content is better separated from frequently queried metadata, and object storage is designed for storing large objects."

Why Redis?

Not simply because:

"Every scalable system needs Redis."

But because:

"Pastebin is read-heavy, and caching frequently accessed pastes can reduce database load and latency."

Why MongoDB?

Not simply because:

"It's NoSQL."

But because:

"Our data is document-oriented and doesn't require complex relational relationships for the core use case."

Conclusion

Designing Pastebin looked simple at first.

But it introduced several system-design concepts that I hadn't thought deeply about before:

  • Database selection
  • Object storage
  • Snowflake IDs
  • Base62 encoding
  • Caching
  • L1/L2 caching
  • CDN
  • Load balancing
  • Rate limiting
  • Expiration
  • Background workers
  • Database replication
  • Database sharding
  • Failure handling
  • Storage trade-offs

More importantly, I'm learning that System Design isn't about memorizing architectures.

It's about being able to explain:

What problem are we solving?

What options do we have?

Why did we choose this one?

What trade-off are we accepting?

That's the skill I'm trying to build — one system at a time. 🚀

Top comments (0)