Database Design • Normalization vs DenormalizationMedium⏱️ ~3 min
Production Fan-out Strategies: Write vs Read Time Materialization
Fan-out-on-write materializes denormalized data when an event occurs. When a user posts, the system immediately writes that post to the feed rows of all their followers. This makes reads trivial: fetch the precomputed feed from one partition. Meta and Pinterest use this for typical users because it keeps read paths Order of magnitude 1 (O(1)) and latency predictable, hitting single digit milliseconds from cache.
The math reveals the challenge. At 2,000 posts per second with an average fan-out of 300 followers, you generate 600,000 derived feed row writes per second. With 3 replicas across regions, that becomes 1.8 million sustained write operations per second. Now consider celebrity accounts: if just 1% of posts come from users with 1 million followers, naive fan-out adds 20,000 posts times 1 million equals 20 billion writes per second in spikes. This is unacceptable and will overwhelm any storage system.
The solution is hybrid fan-out. For high fan-out accounts (celebrities, popular brands), systems switch to fan-out-on-read: store a lightweight pointer and compute the feed when requested, or partially materialize only for active recent followers. Meta special cases accounts with follower counts above thresholds (often 10,000 to 100,000 followers) to avoid write storms. The trade-off is that reads for celebrity content add latency (fetching and merging at request time), but this affects far fewer queries than the write storm would.
Monitoring is critical. Track write amplification (ratio of derived writes to source writes), queue depth during traffic spikes, and p99 write latency. Set explicit thresholds: if a single post triggers more than 10,000 fan-out writes, route it through the read time path. Also measure read latency separately for standard versus celebrity content to ensure your Service Level Objectives (SLOs) hold for both paths.
💡 Key Takeaways
•Fan-out-on-write for average users materializes feed rows immediately: at 2,000 posts per second with 300 average followers and 3 replicas, sustains 1.8 million write operations per second but keeps reads at single digit milliseconds
•Celebrity accounts create write storms: 1% of posts from 1 million follower accounts would generate 20 billion writes per second spikes, requiring fan-out-on-read or partial materialization above thresholds like 10,000 to 100,000 followers
•Hybrid approach trades read latency for write stability: celebrity content adds 30 to 50 milliseconds at read time for merging, but affects fewer queries than a write storm affecting all storage
•Write amplification ratio (derived writes divided by source writes) is the key metric: target staying under 1000x for p99 cases; breaches indicate need for fan-out strategy adjustments
•Partial fan-out optimizes further: materialize only for active recent followers (logged in within 7 days), reducing write volume by 40 to 60 percent while keeping most engaged users on fast path
📌 Examples
Meta Instagram: users with under 10,000 followers use full fan-out-on-write (instant feed updates, 5ms reads); accounts above 100,000 followers switch to fan-out-on-read with merge queries adding 30 to 80 milliseconds but avoiding billions of write operations per celebrity post
Twitter timeline architecture: hybrid model where average users get materialized timelines, but verified accounts with millions of followers store only recent tweets in a hot cache and merge at read time, keeping write throughput under 500,000 operations per second during peaks