How Lambda Architecture Works: The Three Layer System

The Immutable Master Log: Everything starts with an append only event log. Every action in your system (page view, order, sensor reading) gets written to a durable, replayable sequence. Think of this as your source of truth that you can replay years later to reconstruct any state. At a ride sharing company processing 200 thousand events per second at peak, this log might be a distributed commit log with 30 days of retention, storing roughly 500 billion events. This log feeds both processing paths simultaneously. The guarantee you need: if you discover a critical bug in trip revenue calculations six months from now, you can replay the entire log through corrected logic and regenerate all downstream views. The Batch Layer (Cold Path): The batch layer reads events from the master log and writes them into partitioned, immutable storage optimized for large scans. Picture daily or hourly partitions in a data lake, where each partition might contain 50 to 200 GB of compressed data. Batch jobs periodically recompute views from scratch: "for all events between midnight and 11:59pm yesterday, calculate total rides per city, average fare, driver earnings." These jobs are recomputational, not incremental. If today's batch job calculates that San Francisco had 87,432 rides yesterday, that number comes from scanning all raw events for yesterday, not from updating a counter. This makes the logic simple and the results trustworthy. Jobs might take 15 to 45 minutes to process a day of data across a cluster of 100 to 500 compute nodes. The Speed Layer (Hot Path): In parallel, the speed layer consumes the same event stream with minimal delay. It maintains stateful aggregates in memory or low latency storage: current active trips, rides in the last 5 minutes by region, running totals for today. This layer typically keeps only a sliding window, for example the last 6 to 24 hours, because storing unbounded history in fast storage would be prohibitively expensive. The speed layer uses windowing logic that tolerates reasonable out of order arrival. If an event arrives 30 seconds late due to mobile network delays, it still gets counted. State is often maintained in memory with periodic checkpoints to durable storage every 10 to 60 seconds. For a system processing 5 million trips per day, the speed layer might track 50 thousand to 100 thousand active entities at any moment. The Serving Layer: When a dashboard queries "rides per city for today", the serving layer computes: batch result for [midnight, 1am] plus speed result for [1am, now]. The cutoff time (1am in this example) is the latest timestamp for which the batch layer has completed processing. This gives you guaranteed correct historical data plus fresh incremental data. The serving layer often stores batch views in columnar storage optimized for analytical scans and speed views in a low latency key value store indexed by entity and time. Query latency for combined results is typically 50 to 500 milliseconds depending on data volume and query complexity.