Data Processing Patterns • Newsfeed/Timeline Generation (Fan-out Patterns)Medium⏱️ ~2 min
Fanout on Write (Push Pattern) for Timeline Generation
Fanout on write precomputes each follower's home feed immediately when a post is created. When user A posts, the system fetches all of A's followers from the social graph and writes the new post ID into each follower's precomputed timeline cache. This shifts computational cost entirely to the write path, producing exceptionally fast reads that simply retrieve a prebuilt list from memory.
This pattern excels in read heavy workloads where the read to write ratio approaches 100:1. Twitter style systems targeting sub millisecond p50 read latency (around 1 ms from cache, 4 ms p95) rely heavily on this approach for non celebrity users. The precomputed timelines typically store 500 to 800 post IDs per user in sharded in memory caches with 3x replication to minimize tail latency.
The fundamental tradeoff is write amplification. Each post triggers writes proportional to the author's follower count. A user with 10,000 followers generates 10,000 cache writes per post. For celebrities with millions of followers, a single post can trigger millions of fanout writes, creating hotspots that saturate fanout workers, spike queue depths, and blow through network bandwidth budgets across data centers.
Production systems set thresholds (commonly a few thousand followers) beyond which authors are excluded from push fanout. These high degree accounts switch to pull based delivery, where their content is fetched and merged at read time instead. This hybrid approach prevents write storms while preserving fast cached reads for the majority of users.
💡 Key Takeaways
•Optimizes for read heavy workloads with 100:1 read to write ratios; achieves 1 ms p50 and 4 ms p95 cached read latency by eliminating compute on the read path
•Write amplification equals follower count: posting to 10,000 followers generates 10,000 cache writes; scales linearly with social graph out degree
•Memory footprint of 500 to 800 post IDs per user with 3x replication; at tens of KB per user this totals terabytes of RAM across hundreds of millions of users
•Celebrity hotspot failure mode: a single post from an account with 10 million followers triggers 10 million writes, saturating fanout queues and causing propagation Service Level Agreement (SLA) violations beyond 5 seconds
•Hybrid threshold commonly set around a few thousand followers; accounts exceeding this are excluded from push and switched to pull based merging at read time
•Precomputed timelines enable sub 5 second propagation SLAs for normal users but require durable fanout logs and idempotent writes to prevent duplicates under retries
📌 Examples
Twitter style system handling 6,000 writes per second and 300,000 reads per second uses push fanout for 99% of users, storing timelines in Redis clusters sharded by consistent hashing with 3x replication
When a user with 5,000 followers posts, the fanout pipeline batches 5,000 writes across sharded cache nodes, completing propagation in under 2 seconds and hitting the 5 second SLA
Celebrity with 8 million followers is flagged at write time; post skips fanout and is stored in an author specific feed instead, avoiding 8 million writes and preventing queue saturation