Failure Modes and Production Edge Cases

Rebalancing Churn: Kafka Streams relies on the consumer group protocol for task assignment. Every time an instance joins or leaves, a rebalance happens. During rebalance, processing pauses for affected partitions while tasks migrate. In a stable cluster this is fine, but with flaky nodes or aggressive autoscaling, you get frequent rebalances. Each rebalance triggers task migration. If your state stores are large (say 50 to 100 GB per task), restoring that state can take 10 to 20 minutes. During this window, those partitions are effectively offline. If rebalances happen faster than recovery completes, you enter a death spiral: tasks never finish restoring before the next rebalance. The mitigation is sticky assignment (enabled by default) which tries to keep tasks on the same instances, plus standby replicas that maintain hot copies of state stores on backup instances. Standby replicas double your state storage but cut recovery time to seconds. State Store Growth: Windowed aggregations are a classic trap. A 24 hour tumbling window with 1 million unique keys and 500 byte values per window grows to 12 GB per day. Without proper retention settings, state stores balloon. As stores grow, compaction of changelog topics becomes expensive, backup times increase, and recovery slows. You must configure retention to match your actual query needs. If you only serve the most recent hour's data, set state store retention to 2 hours (for safety margin), not infinite. Use key design to limit cardinality. If you're aggregating per user but have 100 million users, consider sampling or hierarchical aggregation. Late Arriving Events: Kafka Streams supports event time processing with configurable grace periods. The grace period defines how late an event can arrive and still update a window. Set it too small, and late events get dropped, causing incorrect aggregates. Set it too large, and you hold windows open longer, increasing memory pressure and delaying results. This is particularly painful with upstream systems that can replay data hours or days late due to batch backfills. If your grace period is 1 hour but events arrive 6 hours late, those events are silently ignored. The only solution is larger grace periods or separate handling for late data. Backpressure and External Sinks: If your topology writes to an external system (database, REST API, cloud storage), that system becomes a bottleneck. Kafka Streams has no built in backpressure mechanism for external sinks. If the sink is slow, your processing slows, consumer lag grows, and in exactly once mode, long running transactions can timeout and abort. Teams handle this by batching writes to external systems, using asynchronous sinks where possible, or decoupling via intermediate Kafka topics (process to Kafka, then use Kafka Connect for final delivery with its own backpressure handling). Version Upgrade Gotchas: Upgrading Kafka broker versions or the Streams library can change consumer group protocol behavior or internal serialization formats. Rolling upgrades must be carefully sequenced. A common issue: mixing Streams application versions during rolling deployment can cause incompatible assignment protocols, leading to split brain where some instances think they own tasks that other instances also own.