What is Kappa Architecture?

Imagine you build a recommendation engine that needs to respond to user clicks within seconds. Six months later, your data scientists improve the algorithm and need to recalculate recommendations for all users based on historical behavior. Traditional systems force you to build two separate pipelines: one for real time (stream processing) and one for historical data (batch processing). This doubles your code, doubles your bugs, and doubles your operational complexity. The Core Problem: Modern products need two things that seem contradictory: low latency decisions (under 2 seconds) AND the ability to recompute results when business logic changes. Classic batch Extract, Transform, Load (ETL) handles recomputation well but delivers results in hours or days. Lambda Architecture tried to solve this by running both batch and stream layers in parallel, but you maintain separate code paths for the same logic. How Kappa Solves This: Kappa uses a single paradigm. Every piece of data is written as an event to an append only log (like a Kafka topic with long retention). One stream processing engine handles everything. For real time needs, it reads from the tail of the log processing fresh events. For recomputation, the same code reads from the beginning of the log, replaying months of history at high speed. When you deploy new logic, you start a new version of your streaming job that replays the historical log while the old version continues serving. Once the new version catches up and passes validation, you switch traffic over. No separate batch code. No reconciliation between two systems. A Concrete Example: An e commerce site writes every user click, search, and purchase to a central event log. A streaming job consumes these events to build user profiles for recommendations, updating within 1 to 2 seconds. When the team ships a new recommendation model needing different features, they deploy a new streaming job that replays 90 days of events from the log at 5 times real time speed, building new profiles in parallel. After catching up, traffic switches to the new profiles. Same code path, same infrastructure.