DEV Community

Cover image for Sharding - Architecture Series: Part 5
MUHAMMAD USMAN AWAN
MUHAMMAD USMAN AWAN

Posted on

Sharding - Architecture Series: Part 5

#webdev #programming #architecture #learning

🏗️ Sharding - Architecture Series: Part 5

⚔️ WHAT is Sharding?

Sharding = Horizontally splitting one huge database into many smaller databases (shards), each living on separate servers.

Each shard stores a slice of the whole dataset and handles a slice of total traffic.

Visual: 

Single DB (Overloaded)       →      Sharded DB (Distributed)
┌─────────────────────┐            ┌───────────┐ ┌───────────┐ ┌───────────┐
│  1TB Data           │            │ Shard 1   │ │ Shard 2   │ │ Shard 3   │
│  15K QPS            │            │ Users A-F │ │ Users G-M │ │ Users N-Z │
│  💥 Slow / Choking  │            │ 3K QPS    │ │ 4K QPS    │ │ 3K QPS    │
└─────────────────────┘            └───────────┘ └───────────┘ └───────────┘
Enter fullscreen mode Exit fullscreen mode

🚨 WHEN Do You Need Sharding? (Red Flags) 

📌 Dataset too large for a single server (100GB–TB scale)
📌 QPS (queries/sec) exceeding hardware limits
📌 One table growing billions of rows
📌 Vertical scaling becomes too expensive 💸
📌 Read/write traffic causing slow queries
Enter fullscreen mode Exit fullscreen mode

When “add more RAM/CPU” stops helping →
It's sharding time.

📱 Real Example: How Instagram Shards

Instagram has 1B+ users, petabytes of posts, reels, feed data.

They shard based on hashed user ID:

shard_id = user_id % 1000
Enter fullscreen mode Exit fullscreen mode

So:

user_id 123456  →  123456 % 1000  →  Shard #456
Enter fullscreen mode Exit fullscreen mode

Everything related to that user (posts, followers, comments) lives on Shard 456, forever.

Why they use hashing? 

✅ Perfect load distribution (no hot shards)
✅ No manual range management
✅ Each user always hits same shard → FAST
Enter fullscreen mode Exit fullscreen mode

⚙️ Sharding Strategies (Choose Your Weapon)

Strategy How It Works Pros Cons
Range ID 1–100K → Shard 1 Easy Hotspots (popular ranges)
Hash (Instagram) ID % N Balanced No range queries
Consistent Hash Hash ring Minimal reshuffling Complex

Quick snippets: 

// Hash Sharding
function getShard(userId) {
  return userId % 1000;
}

// Range Sharding
function getShard(userId) {
  return Math.floor(userId / 100000);
}
Enter fullscreen mode Exit fullscreen mode

🏛️ Architecture (How Apps Route to Shards) 

┌──────────────────────┐     ┌─────────────────┐     ┌────────────┐
│ App Server (API)     │ ───▶│ Shard Router     │ ───▶│ Shard #456 │
│ user_id=123456       │     │ (calculates ID%) │     │ User Data  │
└──────────────────────┘     └─────────────────┘     └────────────┘
Enter fullscreen mode Exit fullscreen mode

If a query needs data from multiple shards → the routing layer handles fan-out + aggregation.

💥 Why Sharding Is So Powerful 

🟢 Linear scalability (add more shards → handle more users)
🟢 Faster queries (smaller DB = faster indexes)
🟢 Fault isolation (Shard 456 down ≠ whole app down)
🟢 Geographic distribution (EU users on EU shards)
🟢 Infinite scaling (theoretically)
Enter fullscreen mode Exit fullscreen mode

This is how Instagram, YouTube, TikTok, Uber handle global scale.

☠️ The Dark Side of Sharding (Things people don’t tell you) 

💀 Cross-shard JOINs = slow and painful
💀 Rebalancing shards = data migration nightmare
💀 Monitoring 1000 shards = complex ops
💀 Schema changes = do it 1000×
💀 Picking the wrong shard key = disaster
Enter fullscreen mode Exit fullscreen mode

Which is why companies denormalize heavily to avoid cross-shard joins.

🎯 Pro Tips from Real Distributed Systems Engineers 

1. Start with 64 or 256 shards, not thousands.
2. Hash your primary keys (best distribution).
3. Never shard on fields that change.
4. Build a routing layer between app ↔ DB.
5. Avoid JOINs across shards — duplicate data instead.
6. Monitor shard imbalance regularly.
7. Plan for re-sharding from day 1.
Enter fullscreen mode Exit fullscreen mode

🚀 Modern Solutions (2025)

These databases handle sharding automatically:

📦 Vitess (YouTube scale)
📦 PlanetScale (MySQL + global)
📦 YugabyteDB (PostgreSQL + distributed)
📦 CockroachDB (ACID + auto-shard)
Enter fullscreen mode Exit fullscreen mode

⭐ Final Summary

Sharding = breaking one big database into many small databases so your system can scale horizontally.

It gives:

  • Infinite scalability
  • Faster queries
  • Better performance
  • Global distribution
  • Instagram/Twitter-level architecture

BUT…

It requires planning, a routing layer, and avoiding cross-shard joins.

🔁 Missed Previous Parts? Catch Up Here!

If you’ve joined this series recently or missed any of the earlier deep-dives, no worries bro — I’ve linked all previous architecture topics below. Each part is designed to build your understanding step-by-step, from caching to replication to sharding. Take your time, go through them in order, and you’ll get a rock-solid grasp of real-world system design fundamentals.

📘 Architecture Series – Index

# Topic
1 Pagination — Architecture Series: Part 1
2 Indexing — Architecture Series: Part 2
3 Virtualization — Architecture Series: Part 3
4 Caching — Architecture Series: Part 4

Top comments (0)