Production Failure Modes: Rebalance Storms, Hotspots, and ISR Shrink

Event streaming systems fail in characteristic ways under load or misconfiguration. Rebalance storms occur when consumer group membership churns: session timeouts trigger rebalances, pausing consumption for 30 to 90 seconds while the coordinator redistributes partitions. If rebalances cascade (slow consumers time out during rebalance, triggering another rebalance), the group never stabilizes and throughput craters. Use cooperative incremental rebalancing (only reassign partitions that need to move) and increase session timeout to 30 to 60 seconds to absorb transient garbage collection pauses. Partition hotspotting arises from poor key distribution: a celebrity user generates 10,000 events per second while typical users generate 1 per minute. The celebrity's partition lags behind, spiking consumer lag from seconds to minutes and blocking downstream joins or aggregations waiting for that partition. Mitigate by salting keys (append a random suffix to spread load) or using composite keys that include time buckets. Monitor per partition throughput and lag; alert on any partition exceeding 2 times the median. ISR shrink and write unavailability occur when followers cannot keep pace with the leader. Network saturation, disk input/output contention, or garbage collection pauses cause followers to fall out of sync. Once ISR drops below the minimum in sync replicas setting (typically 2), quorum writes block entirely, timing out producers. This cascades: blocked producers retry, increasing load, further delaying followers. The mitigation is aggressive producer throttling and backpressure: clients must adapt to broker pushback signals and exponentially back off on errors. Large messages wreck throughput and increase tail latency. A 10 MB message takes 80 milliseconds to transfer at 1 Gbps, blocking the entire partition. Worse, if that message must be redelivered, the consumer pays the full cost again. Store large payloads (images, videos, model weights) in object storage and send pointers in events, keeping messages under 100 KB. Schema evolution breakage is another common failure: a producer deploys a new schema version with a removed field; old consumers crash trying to deserialize. Enforce backward compatibility with schema registries and contract tests before deploying producers.

A social media platform experienced rebalance storms when a deployment caused 10 second garbage collection pauses across all consumers. Session timeout was set to 10 seconds, so every pause triggered a rebalance, which paused other consumers, cascading into continuous rebalancing. Increasing session timeout to 45 seconds stabilized the group.

An e-commerce site keyed order events by user_id. A bulk buyer placed 50,000 orders per day, creating a hot partition that lagged 10 minutes behind others. Downstream join operators timed out waiting for events from that partition. They switched to keying by (user_id, day) to spread the bulk buyer's orders across multiple partitions.

During a network brownout, follower replicas fell 5 seconds behind leaders. ISR dropped to 1 on 20 percent of partitions. Writes with quorum acks blocked, timing out order placement requests. The incident response was to throttle producers by 50 percent using dynamic quotas, allowing followers to catch up in 90 seconds and restore ISR health.

A machine learning pipeline sent 8 MB model weight updates through Kafka. Throughput dropped from 10,000 events per second to 50 per second, and consumer lag spiked to hours. They moved model weights to S3 and sent 200 byte pointers in events, restoring throughput and reducing latency from seconds to milliseconds.