What is Stateful Stream Processing?

The Core Problem: Many real world decisions cannot be made by looking at a single event in isolation. Fraud detection systems need to know "how many transactions has this card made in the last 5 minutes?" Recommendation engines need to track "what did this user click in this session?" Analytics dashboards need to compute "errors per service per minute." None of these questions can be answered by processing one event at a time without memory. Stateless vs Stateful: Stateless processing treats each event independently. You receive an event, transform it, and emit it without remembering anything. This works great for simple filtering or format conversion at 1 to 10 million events per second. Stateful processing maintains context. The system keeps data structures in memory and on disk that track information across events. When a payment event arrives, the processor looks up "how many payments has user 12345 made in the last minute?" updates that counter, and uses the result to decide whether to flag the transaction. How State is Organized: State is partitioned by key and colocated with processing. Events for user 12345 always go to the same processor instance, which maintains state for that user locally. This avoids network calls for every state lookup. The system stores this state in embedded key value stores backed by local Solid State Drives (SSDs), with memory caches on top for fast access, typically 1 to 3 milliseconds per read or write. State Guarantees: The engine must ensure state changes are consistent with event processing, even when machines fail. This means guaranteeing exactly once or effectively once semantics. If a processor crashes after updating state but before checkpointing, the system must either roll back the state update or ensure it does not get duplicated on restart.