Your feed service was reading at 20ms.
Then a celebrity with 2M followers posted.
Now you're at 4 seconds. P99 is on fire.
Here's the setup:
10M users. ~50k posts/day. One account: 2M followers.
Every time that account posts, your feed query fans out across 2M follower rows, sorts by timestamp, and buries your read replicas.
Your team is debating how to fix the delivery model — not the query, the architecture.
A) Fanout on Write — push to every follower's cache on post. Reads instant. Celebrity post = 2M cache writes.
B) Fanout on Read — no precomputation. Fetch + merge at read time. Simple writes. Painful at scale.
C) Hybrid fanout — fanout on write for regular users, fanout on read for celebrities. Merge at read time only for the celebrity slice.
D) Materialized feed table — denormalized per-user feed row, updated async via event stream. One table read, eventual consistency.
Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments.
Top comments (4)
Why D is wrong (Materialized feed table):
Sounds clean — denormalized row per user, updated async via CDC/event stream. But a celebrity post still triggers 2M row updates, just asynchronously. You haven't solved write amplification, you've deferred it. Plus: operational complexity of an event stream + idempotent consumers + reconciliation logic. More moving parts for the same problem.
Why B is wrong (Fanout on Read):
500 accounts followed = 500 queries (or one giant JOIN), sorted in memory, before returning anything. You can cache the result — but now you're doing expensive computation to fill a cache that expires in 30s. That's just a slow fanout-on-write with extra latency. Doesn't survive 10M users.
Why A is the trap (Fanout on Write):
Looks perfect. Reads are instant. Until Cristiano Ronaldo posts and you're writing to 500M caches in parallel. At 2M followers, even 1ms per write = 2,000 seconds of write work. Cache cluster saturates. Write queue backs up. Thundering herd on every celebrity post.
Teams implement this, it works great at 1M users, then hits the celebrity problem when it's already in production. That's the trap.
Answer: C — Hybrid fanout
Why C wins:
This is how Twitter actually solved it — they published the architecture. The insight: fanout-on-write is fast for readers but catastrophic for high-follower accounts. Fanout-on-read is simple but destroys latency at scale. Hybrid is honest about both failure modes.
When a post is created → check follower count. Under 10k? Fan out immediately to all followers' Redis feed caches. Over 10k (celebrity)? Skip the fanout. At read time, fetch from two sources: precomputed cache (regular accounts) + real-time query for celebrity accounts. Merge + deduplicate. The celebrity slice is small and bounded. Result: P99 stays fast, no 2M synchronous writes on celebrity posts.