Partition Stability vs Elasticity: Handling Repartitioning

Modulo Hashing Instability: Using hash(key) mod N to map keys to partitions means changing N remaps many keys to different partitions. With 10 partitions growing to 11, roughly 9 out of 10 keys change partition assignment. This disrupts consumer cache locality and creates thundering herd effects. Consistent Hashing Benefits: Map keys to points on a ring (0 to 2^32) and assign partition ownership to ranges. Adding a partition claims only roughly 1/N of the ring, remapping only that fraction of keys. Virtual nodes (each physical partition owns multiple ring positions) smooth distribution. LinkedIn and Amazon use consistent hashing internally for stable partition assignment. Consumer Rebalance Pain: When partitions are added or consumers join/leave, the group coordinator triggers a rebalance. Naive stop-the-world rebalancing pauses all consumers for seconds to minutes. Cooperative rebalancing and sticky assignment reduce disruption but still add tail latency during transitions. Capacity Planning Best Practices: Minimize resize frequency by starting with headroom and scaling in large increments. Prefer doubling partition count (10 to 20 to 40) rather than incremental adds (10 to 11 to 12). LinkedIn aims to keep partitions under 50% sustained utilization to absorb spikes without emergency resizing.