What is Lambda Architecture?

Lambda Architecture runs two parallel data processing paths to balance speed with accuracy. The speed layer ingests event streams and computes approximate, incremental results in subseconds to a few seconds, giving users fast insights. The batch layer periodically recomputes the exact same metrics from the complete historical dataset every few minutes to hours, correcting errors, removing duplicates, and handling events that arrived out of order. A serving layer merges both outputs so queries get fast results that are eventually corrected by authoritative batch computations. This dual pipeline approach accepts significant operational complexity in exchange for correctness guarantees. You maintain two codebases that implement the same business logic, two separate infrastructures, and reconciliation logic that merges their outputs. Netflix uses this pattern for monitoring pipelines that process millions of events per second, targeting 5 to 30 second latency for operational alerts while batch jobs produce high accuracy daily aggregates for analytics and machine learning training. The fundamental trade off is correctness versus simplicity. Lambda guarantees that batch recomputation will eventually produce the authoritative answer, scrubbing data quality issues and performing complex joins that are impractical in streaming. However, operational costs typically run 1.5 to 2 times higher because both pipelines run continuously, and developers must keep dual logic synchronized to avoid drift where speed and batch outputs diverge.

Netflix runs thousands of streaming jobs for real time monitoring and alerting with 5 to 30 second latency, while batch recompute jobs produce high accuracy hourly and daily aggregates for analytics and ML training datasets

Serving layer uses epoch versioning: speed layer outputs are tagged epoch N, batch recomputation overwrites with epoch N+1, reconciliation logic retracts stale speed records to prevent double counting