Production Scale Reality

The Scale Challenge: At production scale, pre-aggregation and rollup patterns become essential infrastructure, not optional optimization. Consider a large consumer app like Meta with 1 billion monthly active users. The logging pipeline ingests 5 million events per second at peak, generating roughly 400 billion events per day. Storing this raw data in columnar format in object storage is feasible, but querying it directly for dashboards is not. A single complex analytical query scanning even 1 percent of this data might take 10 to 30 seconds and consume hundreds of CPU cores. If you have 50 dashboards refreshed every 5 minutes by various teams, that's 14,400 dashboard loads per day. Running each query against raw data would require thousands of cores running continuously just to serve dashboards, costing millions of dollars per year in compute alone. The Aggregation Layer: Instead, production systems insert an aggregation layer between raw storage and analytical consumers. Batch or streaming jobs continuously compress raw events into metric tables. For example, you might aggregate per user, per app version, per country, per hour. With 1 billion users, 100 app versions, 200 countries, and 24 hours, that's approximately 480 billion combinations theoretically. In practice, you aggregate only active combinations, reducing data volume by 100x to 1000x. Now the online analytics service powering dashboards hits these pre-aggregated tables. With reduced data size and better partitioning, queries run in 50 to 300 milliseconds at p50 and under 1 second at p99, even at several thousand queries per second. This is how internal BI tools and executive dashboards achieve interactive performance on petabyte scale datasets. Hybrid Architectures: Large systems often use hybrid approaches. Streaming aggregation maintains metrics for the last 24 to 48 hours in a low latency store with 10 to 60 second freshness. Batch pipelines periodically compact these into longer term tables optimized for historical queries. This splits the workload: streaming handles operational monitoring where freshness matters, batch handles heavy analytical queries where 1 to 24 hour lag is acceptable. Cost Impact: The economics are compelling. At Meta scale, pre-aggregation reduces compute cost for analytical queries by 90 to 95 percent compared to always querying raw data. Storage costs increase by perhaps 30 to 50 percent, but compute is typically 5 to 10 times more expensive per unit of work, making this a clear win. The system complexity is the real cost: managing aggregation pipelines, ensuring correctness, and handling backfills requires significant engineering investment.