How Modern Warehouse Architecture Works at Scale

The End to End Flow: At FAANG scale, you have hundreds of microservices each generating thousands of writes per second to their own OLTP databases, plus event streams emitting 100k to 1M events per second. The warehouse needs to ingest this data, transform it, and make it queryable within minutes, all without impacting production systems. Change Data Capture (CDC) for OLTP Sources: Running SELECT * FROM users every hour on a 100 million row table would crush your production database. Instead, CDC pipelines monitor the database transaction log and capture only inserts, updates, and deletes as they happen. The pipeline publishes these changes to a message queue, then writes them to the data lake as partitioned files (typically partitioned by date and hour). End to end lag from database commit to data lake landing is typically under 1 minute. This gives you four nines durability in the lake without putting read load on OLTP systems. Streaming Ingestion for Events: User clicks, application logs, and IoT telemetry arrive as event streams. A streaming ingestion layer consumes from message queues like Kafka, batches events into files every few minutes, and writes them to the lake partitioned by event time. At companies like Netflix, this might mean ingesting 50 TB daily from click streams alone. Medallion Transformation Pattern: Raw data lands as bronze tables: immutable, append only files with exactly the schema from the source. A transformation layer reads bronze, applies cleaning and normalization, and writes silver tables with conformed schemas. Finally, it computes business metrics and aggregates into gold tables optimized for specific use cases. At Uber scale, daily transformation jobs scan tens of petabytes but finish within 1 to 3 hour Service Level Agreements (SLAs) by distributing work across thousands of compute cores. The key is that compute can scale horizontally: add more workers to process more data in the same time. Serving with Elastic Compute: The serving layer runs a distributed SQL engine that reads from curated gold tables. When 500 analysts arrive at 9am and start querying, the system automatically spins up more compute nodes. When load drops in the evening, it scales back down. This elasticity is the breakthrough: you pay for compute only when you use it, and you can handle spikes without manual intervention. Production dashboards query with 10 to 30 second timeouts. Interactive exploration targets sub second to few second latency by leveraging result caching, materialized views of common aggregates, and columnar storage that only reads the columns you need.