There is a kind of outage that only happens to social products, and only on a good day. Traffic is up, a post is spreading, someone with a real audience just joined — and the feed, fine for eight months through every load test, starts taking four seconds, then eight, then times out.
Nothing broke. The architecture did what it was designed to do. It was designed for a graph that no longer exists.
Here's how feeds get assembled, why both textbook approaches are wrong alone, and what "design for the hybrid, ship the simple one" means in code. The commercial framing lives in our breakdown of social media app development cost; this one is for whoever implements it.
The two primitives
Fanout on read stores posts once and builds the feed on request.
GET /feed
followees = graph.following(user_id) # N ids
posts = posts.recent_for(followees, limit=200)
return merge_sort_by_time(posts)[:50]
One row per write, one copy in storage, correct by construction — no derived state, no backfill, no repair tooling. The cost sits on the read path, proportional to |following| × posting_rate: at 30 followees it's a scatter-gather over a warm index you'll never think about; at 3,000 it's a 3,000-way fan-in on every refresh.
Fanout on write inverts it: on post, push a reference into a materialized timeline for every follower.
POST /post
post_id = posts.insert(...)
for follower_id in graph.followers(author_id): # M ids
timelines.push(follower_id, post_id)
Reads become one range scan — instant and boringly predictable, which is what you want on the path 95% of traffic takes. The cost moved to writes, proportional to |followers|: dozens for a normal user, half a million for a 500k-follower account. At five million, you've built a self-inflicted DDoS with a compose button.
Why the naive version demos perfectly
Fanout on read isn't dumb — it's the right v1, which is what makes it dangerous. With a hundred seeded test users it's indistinguishable from a world-class architecture; every signal is positive. What you're missing is that its performance depends on a variable you don't control and haven't measured: the shape of your social graph.
Graph shape doesn't drift gently — it's power-law distributed. Your p50 user follows 40 accounts and always will; your p99 follows 2,000. Then a large account joins, and a slice of your users share a followee whose posting rate alone saturates the merge step. That's a cliff, not a slope, and it correlates with success, so it arrives when you have the least slack.
The expense isn't the code; write-time fanout is a few hundred lines. It's doing it live: dual-writing while you backfill, discovering the backfill can't outrun the write rate it competes with, shipping a repair job for the timelines you corrupted at 2am on attempt one.
Every real platform ends up hybrid
Nobody operating a large feed picked one. They converged on the same shape, because the trade-off flips at a follower threshold.
- Below the threshold → fanout on write. Cheap writes, instant reads.
- Above it → no fanout. Those posts stay in a pull-only store.
- At read time: load the materialized timeline, query the small set of large accounts this user follows, merge, rank.
GET /feed
base = timelines.range(user_id, cursor, 50) # precomputed
celebs = graph.following_celebrities(user_id) # usually < 20
hot = posts.recent_for(celebs, since=cursor) # small fan-in
return rank(merge(base, hot))[:50]
The distribution works both ways: almost all accounts are small, so almost all writes fan out cheaply; almost all users follow only a handful of large accounts, so the merge stays bounded. You pay each cost only where it's small.
The parts worth senior time:
The threshold isn't a constant. Follower count is a proxy; what matters is followers × posting_rate. A moderate account posting 40 times a day hurts more than a huge one posting weekly.
Crossing it is a state machine. An account grows past the line — drain its timeline entries or leave them? Gets demoted? You now hold partially-materialized timelines. Store provenance; you'll want it mid-incident.
Merge boundaries are where correctness dies. Two streams, two clocks, one cursor. Stable pagination across a merged read — no duplicates, no drops, nothing reappearing three scrolls later — is fiddlier than the fanout itself.
Timelines are a cache, so treat them like one. Bounded length, TTL for inactive users, working rebuild path. If you can't reconstruct one from source of truth on demand, that's not a cache — it's a second database with no backups.
What to actually do on day one
Not "build the hybrid." Building the sophisticated version first is its own failure mode: you'll guess your graph shape wrong, and unwinding a bad write-time fanout is worse than replacing a naive read.
- Ship fanout on read. Correct v1, correct by construction.
-
Put it behind one
FeedAssemblerseam —assemble(user_id, cursor) -> [PostRef]. One seam, not one per call site. That interface is the entire cost of your future migration. - Instrument the graph, not just latency. Distribution of following count, of follower count, posting rate by follower bucket, read-to-write ratio. Your APM won't give you these.
- Set the trigger in advance, while calm: "when p99 assembly crosses X ms, or any account crosses Y followers, we start the hybrid."
- Write the dual-write and backfill plan before you need it. Not the code — the plan. The worst version of this project is the one designed under load.
Designing for the hybrid and deferring it costs an interface and a dashboard. Not designing for it costs two quarters, when you can least afford them.
The ranking wrinkle
Ranked feeds used to need a data moat — collaborative filtering wants real interaction history before it beats reverse-chronological. Embedding-based retrieval moves that cold-start line: an interest-matched feed from content semantics plus thin engagement signals. Our notes on LLM integration architecture cover wiring those layers up, and we're happy to look at your graph before you commit to a model.
Frequently Asked Questions
At how many followers should an account switch to pull-only in a hybrid feed?
There's no universal number, and anyone quoting one hasn't seen your graph. Starting points often sit in the tens of thousands, but the metric that matters is followers × posting_rate, not followers alone. Make the threshold runtime config, log fanout cost per author for a few weeks, then set it where your write-path p99 starts suffering.
Can I use fanout-on-read forever if my app is a small niche community?
Possibly, and it may be the correct call. If your median user follows a few dozen accounts and nobody ever exceeds a few thousand followers, read-time assembly on a decent index will serve you for years. The risk isn't community size — it's whether a single high-follower account can ever appear.
How do you handle deletes and blocks with a write-time fanout timeline?
Most teams use both approaches. Eager removal fans the delete out like the post: expensive, keeps timelines clean. Lazy filtering leaves the reference and filters at read time against a block/delete set: cheap writes, extra check on the hottest path. Blocks usually want lazy filtering; hard deletes want eager removal plus a lazy filter as a safety net, because a deleted post resurfacing is a trust incident, not a bug.
Does a hybrid feed need Redis or Kafka specifically?
No — the pattern is stack-agnostic. You need per-user ordered lists with cheap appends and range reads (Redis sorted sets, a wide-column store, even a well-indexed relational table at modest scale) plus something to run fanout off the write path. Picking exotic infrastructure before measuring your graph is how teams end up operating a Kafka cluster for 4,000 users.
How do I test this failure mode before production?
Synthesize the graph you fear, not the one you have: seed an account with the follower count you expect your largest to reach, give it a realistic posting rate, and load-test against that rather than the uniformly-distributed fake users most fixtures generate. If your fixture has no power-law tail, it isn't testing your feed.
Is the feed the biggest cost driver in a social build, or is moderation?
Feed architecture is the biggest build and infrastructure swing; moderation used to be the biggest operating cost. That second one changed once classification pipelines could triage most content automatically, shifting moderation from unbounded headcount to a mostly-fixed engineering line. See the full cost breakdown for both sides.
Top comments (0)