Data Processing Patterns • Newsfeed/Timeline Generation (Fan-out Patterns)Medium⏱️ ~2 min
Fanout on Read (Pull Pattern) for Timeline Generation
Fanout on read assembles a user's timeline on demand at read time by fetching recent posts from each account they follow, then merging and ranking the results. When user B requests their home feed, the system expands B's follow list, queries per author stores or a global recent index for the latest posts from each followee, applies privacy filters (blocks, mutes), and runs ranking algorithms to produce a personalized page. All computation happens on the read path.
This pattern eliminates write amplification entirely. Posting is a constant time operation that writes once to the author's own feed, regardless of follower count. A celebrity with 10 million followers incurs the same write cost as a user with 10 followers. This makes pull based systems ideal for handling high degree nodes and avoiding write hotspots that plague push systems. Facebook's Multifeed architecture exemplifies this approach, prioritizing relevance and personalization over strict chronological ordering.
The tradeoff is read amplification and latency. Generating a feed for a user following 1,000 accounts requires fetching and merging 1,000 recent post streams. At peak traffic (hundreds of thousands of Queries Per Second (QPS)), this multiplies compute load and can push read latency beyond acceptable bounds, especially when deep ranking models are applied. Pull systems must carefully cache merged results and limit candidate set sizes to meet Service Level Objectives (SLOs) like sub 2 second feed retrieval.
Pull works best when users follow many accounts, when personalization depth matters more than instant propagation, and when inactive users dominate (no wasted fanout work). The pattern scales write throughput naturally but requires robust caching, efficient graph expansion, and lightweight ranking to control read path costs.
💡 Key Takeaways
•Eliminates write amplification: posting is constant time regardless of follower count; celebrity with 10 million followers writes once, same cost as user with 10 followers
•Read amplification scales with follow count: user following 1,000 accounts triggers 1,000 per author fetches plus merge and ranking; compute cost concentrates on read path
•Ideal for dense social graphs and personalization: Facebook Multifeed uses pull to apply Machine Learning (ML) ranking over large candidate sets, trading read latency for relevance
•Read latency challenges at scale: assembling feeds for users following thousands of accounts can exceed 2 second SLOs; requires aggressive caching of merged results and candidate set limits
•Thundering herd failure mode: synchronized reads during breaking news cause Central Processing Unit (CPU) spikes from parallel graph expansions; mitigated by per user home cache and request coalescing
•Best for inactive user bases: avoids wasting fanout work on users who rarely read; pull only computes when user actually requests feed, saving resources
📌 Examples
Facebook Multifeed expands user's social graph at read time, fetches recent actions from friends and pages, applies ML ranking for relevance, and serves results within 2 second SLO by caching intermediate candidate sets
User following 500 accounts requests feed: system fetches top 20 recent posts from each author (10,000 candidates), applies privacy filters, ranks by affinity score, and returns top 50 in under 1.5 seconds
Celebrity posts once to own feed (single write); 10 million followers who read later each trigger a pull that fetches and merges celebrity's recent posts with other followees, distributing read cost over time