DEV Community

Cover image for HLD: Instagram Feed
Gouranga Das Samrat
Gouranga Das Samrat

Posted on

HLD: Instagram Feed

#systemdesign #backend #algorithms

Design a photo-sharing platform where users post photos and see a feed of photos from people they follow.

1️⃣ Clarify Requirements

Functional Requirements

  • Upload photos/videos
  • Follow/unfollow users
  • View home feed (photos from followed users, newest first)
  • Like, comment on photos
  • User profile with photo grid

Non-Functional Requirements

  • High availability
  • Eventual consistency for feed (seconds lag is fine)
  • Low latency feed load < 200ms
  • Image-heavy — media storage is the core challenge
  • Scale: 500M DAU, 100M photos uploaded/day

2️⃣ Estimate Scale 

Photo uploads/day: 100M
Write QPS: 100M / 86,400 ≈ 1,160 uploads/sec
Read QPS: 500M × 10 feed loads/day / 86,400 ≈ 58,000 reads/sec

Storage:
  Avg photo: 3 MB (original) + thumbnails (100KB, 500KB) ≈ 4 MB total
  100M photos/day × 4 MB = 400 TB/day
  5-year storage: ~730 PB (need tiered storage!)

CDN bandwidth:
  58K reads/sec × 10 photos/feed × 500KB avg = ~290 GB/sec outbound
Enter fullscreen mode Exit fullscreen mode

3️⃣ Architecture Overview 

[Client]
   │
[CDN] ← photos served here (not from origin)
   │
[API Gateway + LB]
   │
   ├── [Photo Service] → upload → [S3] → trigger → [Resize Worker]
   │                                                      → S3 (thumbnail, medium, large)
   │                                                      → Update metadata DB
   │
   ├── [Feed Service] → Redis (pre-computed feed) → return photoIds → CDN URLs
   │
   ├── [User Service] → PostgreSQL
   │
   └── [Social Graph] → Cassandra (followers/following)
Enter fullscreen mode Exit fullscreen mode

4️⃣ Photo Upload Flow 

1. Client → POST /upload → Photo Service
2. Photo Service → generate uploadId → return S3 pre-signed URL
3. Client → upload directly to S3 (bypasses your servers!)
4. S3 → triggers Lambda / sends event to queue
5. Resize Worker → create 3 sizes → store in S3
6. Update metadata DB: photoId, userId, S3 keys, created_at
7. Fan-out → push photoId to followers' feed caches (async)
Enter fullscreen mode Exit fullscreen mode

Pre-signed URL pattern = your servers never touch the image bytes.

5️⃣ Feed Generation (Hybrid Fan-out)

Same pattern as Twitter:

  • Regular users → fan-out on write
  • Celebrities → fan-out on read, merge at read time 
Redis feed cache:
  Key: feed:{userId}
  Value: Sorted Set of {photoId, timestamp}
  Size: 500 entries max
  TTL: 7 days
Enter fullscreen mode Exit fullscreen mode

6️⃣ Database Schema 

-- Photos
CREATE TABLE photos (
  photo_id    BIGINT PRIMARY KEY,  -- Snowflake ID
  user_id     BIGINT NOT NULL,
  s3_key_orig VARCHAR(500),
  s3_key_med  VARCHAR(500),
  s3_key_thumb VARCHAR(500),
  caption     TEXT,
  created_at  TIMESTAMP DEFAULT NOW(),
  likes_count BIGINT DEFAULT 0
);

-- Social graph (Cassandra)
CREATE TABLE following (
  follower_id BIGINT,
  followed_id BIGINT,
  PRIMARY KEY (follower_id, followed_id)
);
Enter fullscreen mode Exit fullscreen mode

7️⃣ Key Challenges

Storage Tiering 

Hot (0-7 days):   S3 Standard (fast, expensive)
Warm (7-90 days): S3 Infrequent Access (cheaper)
Cold (90+ days):  S3 Glacier (very cheap, slow retrieval)
Enter fullscreen mode Exit fullscreen mode

Serving Photos

  • NEVER serve from origin → always from CDN
  • URL format: https://cdn.instagram.com/photos/{photoId}/{size}.jpg
  • Immutable URLs (content never changes for same ID) → cache forever

✅ Trade-offs Summary

Decision Choice Rationale
Photo storage S3 + CDN Scalable object store, edge serving
Upload pattern Pre-signed URL Don't bottleneck servers with bytes
Feed Hybrid fan-out Celebrities break pure push
Social graph Cassandra Write-heavy, large-scale relationships
Image sizes 3 variants Different contexts (thumbnail, feed, full)

Top comments (0)