Stateful Aggregation: How Windowing Actually Works

The Architecture: Windowing is implemented using stateful operators partitioned by key. Messages are routed so all events for a specific key (user ID, card number, service name) go to the same processing task. Each task maintains a local state store holding partial aggregates per window per key. Think of it like a distributed hash map where the key is "user123, window 12:00 to 12:05" and the value is the running aggregate (count of 47, sum of 1283.50). When a new event arrives, the engine looks up the relevant window state, updates it, and decides whether to emit results. State Management at Scale: At 1 million events per second with millions of active keys, state size becomes critical. Systems typically cap state to tens of gigabytes per task using a combination of in memory caches and persistent backing stores like RocksDB. Checkpoint intervals matter enormously. With 30 to 60 second checkpoints, recovery from failure takes under 2 minutes. But longer checkpoints or larger state means slower recovery, potentially violating Service Level Agreements (SLAs). Update Semantics: When late events arrive after a window has closed, the engine can emit retractions or updates. The downstream store must handle these correctly. Some systems send a negative record to cancel the old aggregate, then a positive record with the corrected value. Others support upserts where the new value replaces the old. Production systems often pair streaming with batch recomputation. A nightly batch job recomputes the last 24 hours of aggregates from durable logs and compares with streaming results. Differences beyond a small tolerance indicate configuration issues, giving strong correctness guarantees while retaining low latency benefits.