State Management and Checkpointing at Scale

State Storage Architecture: Each processing task maintains local state in an embedded key value store, typically RocksDB, backed by local Solid State Drives (SSDs). Above this sits an in memory cache for hot keys. This two tier approach provides 1 to 3 millisecond read and write latency without network calls. For a system processing 1 million events per second across 100 tasks, each task might hold 5 to 20 gigabytes of physical state on disk, with several hundred megabytes cached in memory. The total logical state can be enormous. A user behavior pipeline at a large consumer company might maintain 30 days of per user aggregates across 100 million users. At 1 kilobyte per user, that is 100 gigabytes of state, spread across hundreds of machines. The key insight is that state is partitioned by key: events for user 12345 always route to the same task, which holds only that user's state locally. The Checkpointing Mechanism: Checkpointing ties input offsets to state snapshots. At checkpoint time, the system picks a consistent set of input positions across all partitions, flushes in memory state to the local store, and records metadata mapping partitions and offsets to state files. Here is what happens during a checkpoint: Checkpoint frequency is a trade off. Frequent checkpoints (every 10 seconds) mean fast recovery: if a task fails, it only replays 10 seconds of input. But frequent checkpoints add overhead. Flushing gigabytes of state to disk every 10 seconds can saturate I/O. Less frequent checkpoints (every 5 minutes) reduce overhead but increase recovery time: a failure means replaying 5 minutes of events. Changelogs and Recovery: Many systems also write state changes to a distributed log called a changelog. Every update to local state gets appended to this log. On recovery, a task can restore state by replaying the changelog instead of reprocessing the entire input stream. This is faster: replaying a changelog is a sequential read of compact state updates, while replaying input might mean filtering and aggregating millions of events. For high availability, some systems maintain warm standbys that continuously consume the changelog, keeping a near real time replica of state. When the primary fails, the standby can take over in seconds with minimal replay, achieving recovery times under 10 seconds instead of minutes.